94
Accountability in Developing Assessment Tools Carlo Magno MCL Consultant

Accountability in developing assessment tools

Embed Size (px)

Citation preview

Page 1: Accountability in developing assessment tools

Accountability in Developing Assessment Tools

Carlo Magno

MCL Consultant

Page 2: Accountability in developing assessment tools

Objectives

• 1. Check the alignment between the objectives set in the syllabus and the items contained in the assessment.

• 2. Distinguish in using paper and pencil or performance-based needed when creating assessment tasks.

• 3. Follow the rules in writing types of items and performance-based tasks.

Page 3: Accountability in developing assessment tools

Assessment Process

Objectives in the syllabus

Create Assessment

tasks

Content Validity Review

assessment tasks

Revision

Data Gathering Data Analysis

Teacher Teacher

Teacher

Teacher

Teacher

Supervisor

•Needs of students and society •Mission and vision of the school •National and International Standards •Existing programs

Page 4: Accountability in developing assessment tools

What to check? C

on

ten

t V

alid

ity Check for alignment between the course

objectives and the competency covered in the task

Check if the assessment is appropriate with the form assessment task made (paper and pen vs. performance-based)

Face

V

alid

ity

Minor: Rules in writing the tasks

Minor: Syntax (grammar)

Minor: Suitability for learners

Page 5: Accountability in developing assessment tools

Task!

• Objective: Prepare a financial report

• Assessment task?

• Objective: Create a promotional material for the Chocolate Hills

• Assessment Task?

• Objective: calculate the total assessable benefits of an employee

• Assessment task?

Page 6: Accountability in developing assessment tools

Task!

• Objective: Interpret information obtained from a radar

• Assessment task:

• Objective: Give the advantage of multithreading.

Page 7: Accountability in developing assessment tools

Task!

• Objective: Classify whether the building pictures belongs to traditional, industrial and post-industrial style.

• Assessment Task:?? • Objective: Illustrate Maxwell's Theorem of

Reciprocal Displacements. • Assessment task:?? • Objective: Graphs a circle and other geometric

figures on the coordinate plane • Assessment Task:

Page 8: Accountability in developing assessment tools

Task

• Objective: Compare accommodation theory and face negotiation theory.

• Assessment task?

• Objective: Classify soluble and insoluble compounds

• Assessment task?

Page 9: Accountability in developing assessment tools

V. Matching Items with Criterion Item

• Instruction: Indicate whether the matching of the objective and item is suitable.

• 1. Objective: Given a performance of an instrumental or vocal melody containing a melodic or rhythmic error, and given the score for the melody, be able to point out the error.

• Criterion item: The instructor will play the melody of the attached musical score on the piano and will make an error either in rhythm or melody. Raise your hand when the error occurs.

• Is the item Suitable? ___ Yes

____ No ____Can’t tell

Page 10: Accountability in developing assessment tools

V. Matching Items with Criterion Item

• 2. Objective: Given mathematical equations containing one unknown, be able to solve for the unknown.

• Criterion Item: Sam weighs 97 kilos. He weighs 3.5 kilos more than Barrey. How much does Barry weigh?

• Is the item Suitable? ___ Yes

____ No

____Can’t tell

Page 11: Accountability in developing assessment tools

V. Matching Items with Criterion Item

• 3. Objective: Be able to demonstrate familiarity with sexual anatomy and physiology

• Criterion Item: Draw and label a sketch of the male and female reproductive systems.

• Is the item Suitable? ___ Yes

____ No

____Can’t tell

Page 12: Accountability in developing assessment tools

V. Matching Items with Criterion Item

• 4. Objective: Given any one of the computers in our product line, in its original carton, be able to install and adjust the machine, preparing it for use. Criteria: The machine shows normal indication, and the area is free of debris and cartons

• Criterion item: Select one of the cartons containing one of

our model XX computers, and install it for the secretary in Room 45. Make sure it is ready for use and the area is left clean.

• Is the item Suitable? ___ Yes ____ No ____Can’t tell

Page 13: Accountability in developing assessment tools

V. Matching Items with Criterion Item

• 5. Objective: When given a set of paragraphs (that use words within your vocabulary), some of which are missing topic sentences, be able to identify the paragraph without topic sentences.

• Criterion Item: Turn to page 29 in your copy of Silas

Marner. Underline the topic sentence of each paragraph on that page.

• Is the item Suitable? ___ Yes ____ No ____Can’t tell

Page 14: Accountability in developing assessment tools

Observe the two sets of objectives

• Sequence the evolution of different OS

• Recognize specific conditions that will put people at risk in a building

• Select from the list what information is useful when performing a fitness assessment

• Convert moles to kilograms

• Create a program that will compute the grades of students using java.

• Construct a pole that can supporting a 30 sq flat wood on top.

• Demonstrate how to convince tourist to visit the old church.

• Sell a new kind of coffee among customers in a coffee shop.

Page 15: Accountability in developing assessment tools

Paper and pen vs. Performance-based tasks

• Format: binary, multiple choice, matching type, short answer

• Measure discreet skills (cognitive)

• Group administration

• Format: demonstrations, performances, actual work (authentic)

• Measure complex integrated skills (cognitive, psychomotor, affect)

• Individual or several groups

Page 16: Accountability in developing assessment tools

Constructive Alignment Objective: Interpret a table showing the population density of

the world (Understanding) Test Item: Given a bar graph, report the years where the

population increased and decreased, describe the pattern of growth each year and provide possible explanations for the growth pattern.

Objective: Draw a picture using the principle of balance. (Creating,)

Performance task: Draw a picture in the box showing the principle of balance.

Objective: Recognize fallacies in an argument (Understanding) Test item: Read the given article and list down five fallacious

statements found. Objective: Run in a 100-yard dash (psychomotor) Performance Assessment: Students need to complete a run in

100 yard dash.

Page 17: Accountability in developing assessment tools

Determine if the objectives will require paper and pen or performance-based task

• 1. State the domain and range of a given function • 2. Solve systems of linear equations in two variables by the

graphical method • 3. Draw the slope of the given equation • 4. Simplify complex fractions • 5. Deliver a speech to persuade people to donate for the

orphanage • 6. Multiply and divide rational algebraic expression • 7. List hazards of working with chemicals • 8. Classify the properties of substances as to physical or chemical

and extrinsic or intrinsic • 9. Cite evidence of chemical changes • 10. Present ways to prevent walls from easily cracking during

earthquakes

Page 18: Accountability in developing assessment tools

Paper and pencil assessment

• Binary type – remembering - understanding

• Matching type - remembering

• Multiple choice – remembering, understanding, applying, analyzing, evaluating

• Supply type - remembering

Page 19: Accountability in developing assessment tools

Binary Type

• 1. Avoid the use of “specific determiners”

• FAULTY: No picture-no sound in a television set may indicate a bad 5U4G.

• IMPROVED: A bad 5U4G tube in a television set will result in no picture sound.

Why is the item faulty?

Page 20: Accountability in developing assessment tools

Binary Type

• 2. Base true-false items upon statements that are absolutely true or false, without qualifications or exceptions.

• FAULTY: World War II was fought in Europe and the Far East.

• IMPROVED: The primary combat locations in terms of military personnel during World War II were Europe and the Far East.

Why is the item faulty?

Page 21: Accountability in developing assessment tools

Binary Type

• 3. Avoid negative stated items when possible and eliminate all double negatives.

• FAULTY: It is not frequently observed that copper turns green as a result of oxidation.

• IMPROVED: Copper will turn green upon oxidizing.

Why is the item faulty?

Page 22: Accountability in developing assessment tools

Binary Type

• 4. Use quantitative and precise rather than qualitative language where possible.

• FAULTY: Many people voted for Gloria Arroyo in the 2003 Presidential election.

• IMPROVED: Gloria Arroyo received more than 60 percent of the popular votes cast in the Presidential election of 2003.

Why is the item faulty?

Page 23: Accountability in developing assessment tools

Binary Type

• 5. Avoid stereotypic and textbook statements.

• FAULTY: From time to time efforts have been made to explode the notion that there may be a cause-and-effect relationship between arboreal life and primate anatomy.

• IMPROVED: There is a known relationship between primate anatomy and arboreal life.

Why is the item faulty?

Page 24: Accountability in developing assessment tools

Binary Type

• 6. Avoid making the true items consistently longer than the false items.

• 7. Avoid the use of unfamiliar or esoteric language. • FAULTY: According to some peripatetic politicos, the

raison d’etre for capital punishment is retribution. • IMPROVED: According to some politicians,

justification for the existence of capital punishment can be traced to the Biblical statement, “An eye for an eye.”

Why is the item faulty?

Page 25: Accountability in developing assessment tools

Binary Type

• 8. Avoid complex sentences with many dependent clauses.

• FAULTY: Jane Austen, an American novelist born in 1790, was a prolific writer and is best known for her novel Pride and Prejudice, which was published in 1820.

• IMPROVED: Jane Austen is best known for her novel Pride and prejudice.

Why is the item faulty?

Page 26: Accountability in developing assessment tools

Binary Type

• 9. It is suggested that the crucial elements of an item be placed at the end of the statement.

• FAULTY: Oxygen reduction occurs more readily because carbon monoxide combines with hemoglobin faster than oxygen does.

• IMPROVED: Carbon monoxide poisoning occurs because carbon monoxide dissolves delicate lung tissue.

Why is the item faulty?

Page 27: Accountability in developing assessment tools

Matching Type Match List A with List B. You will be given one point

for each correct match. List A List B a. cotton gin a. Eli Whitney b. reaper b. Alexander Graham Bell c. wheel c. David Brinkley d. TU54G tube d. Louisa May Alcott e. steamboat e. None of these

•Directions failed to specify the basis for matching •List are enumerated identically •Responses not listed logically •Lacks homogeneity •Equal number of elements •Use of “None of the above”

Why is the item faulty?

Page 28: Accountability in developing assessment tools

Matching Type

IMPROVED: Famous inventions are listed in the left-hand column and inventors in the right-hand column below. Place the letter corresponding to the inventor in the space next to the invention for which he s famous. Each match is worth 1 point, and “None of these” may be the correct answer. Inventors may be used more than once.

Inventions Inventors __ 1. steamboat a. Alexander Graham-Bell __ 2. cotton skin b. Robert Fulton __ 3. sewing machine c. Elias Howe __ 4. reaper d. Cyrus McCormick e. Eli Whitney

Page 29: Accountability in developing assessment tools

Multiple Choice

1. Salvador Dali is

a. a famous Indian.

b. important in international law.

c. known for his surrealistic art.

d. the author of many avant-garde plays.

•It is recommended that the stem be a direct question. •The stem should pose a clear, define, explicit, and singular problem.

Why is the item faulty?

Page 30: Accountability in developing assessment tools

Multiple Choice

IMPROVED: With which one of the fine arts is Salvador Dali associated?

a. surrealistic painting

b. avant-garde theatre

c. polytonal symphonic music

d. impressionistic poetry

Page 31: Accountability in developing assessment tools

Multiple Choice

2. Milk can be pasteurized at home by

a. heating it to a temperature of 130o

b. Heating it to a temperature of 145o

c. Heating it to a temperature of 160o

d. Heating it to a temperature of 175o

•Include in the stem any words that might otherwise be repeated in each response.

Why is the item faulty?

Page 32: Accountability in developing assessment tools

Multiple Choice

IMPROVED: The minimum temperature that can be used to pasteurize milk at home is:

a. 130oC

b. 145oC

c. 160oC

d. 175oC

Page 33: Accountability in developing assessment tools

Multiple Choice 3. Although the experimental research, particularly that by

Hansmocker must be considered equivocal and assumptions viewed as too restrictive, most testing experts would recommend as the easiest method of significantly improving paper-and-pencil achievement test reliability to

a. increase the size of the group being tested.

b. increase the differential weighting of items.

c. increase the objective of scoring.

d. increase the number of items.

e. increase the amount of testing time.

Items should be stated simply and understandably, excluding all nonfunctional words from stem and alternatives.

Why is the item faulty?

Page 34: Accountability in developing assessment tools

Multiple Choice

IMPROVED: Assume a 10-item, 10-minute paper-and-pencil multiple choice achievement test has a reliability of .40. The easiest way of increasing the reliability to .80 would be to increased

a. group size

b. scoring objectivity

c. differential item scoring weights

d. the number of items

e. testing time

Page 35: Accountability in developing assessment tools

Multiple Choice

4. None of the following cities is a state capital except

a. Bangor

b. Los Angeles

c. Denver

d. New Haven

•Avoid negatively stated items

Why is the item faulty?

Page 36: Accountability in developing assessment tools

Multiple Choice

IMPROVED: Which of the following cities is a state capital?

a. Bangor

b. Los Angeles

c. Denver

d. New Haven

Page 37: Accountability in developing assessment tools

Multiple Choice

5. Who wrote Harry Potter and the Goblet of Fire?

a. J. K. Rowling

b. Manny Paquiao

c. Lea Salonga

d. Mark Twain

•If possible the alternatives should be presented in some logical, numerical, or systematic order. •Response alternatives should be mutually exclusive.

Why is the item faulty?

Page 38: Accountability in developing assessment tools

Multiple Choice

IMPROVED: Who wrote Penrod?

a. J. K. Rowling

b. J. R. R. Tolkien

c. V. Hugo

d. L. Carrol

Page 39: Accountability in developing assessment tools

Multiple Choice

6. Which of the following statements makes clear the meaning of the word “electron”?

a. An electronic tool

b. Neutral particles

c. Negative particles

d. A voting machine

e. The nuclei of atoms

•Make all responses plausible and attractive to the less knowledgeable and skillful student.

Why is the item faulty?

Page 40: Accountability in developing assessment tools

Multiple Choice

IMPROVED: Which of the following phrases is a description of an “electron”?

a. Neutral particle

b. Negative particle

c. Neutralized proton

d. Radiated particle

e. Atom nucleus

Page 41: Accountability in developing assessment tools

Multiple Choice

7. What is the area of a right triangle whose sides adjacent to the right angle are 4 inches long respectively?

a. 7

b. 12

c. 25

d. None of the above

•The response alternative “None of the above” should be used with caution, if at all.

Why is the item faulty?

Page 42: Accountability in developing assessment tools

Multiple Choice

IMPROVED: What is the area of a right triangle whose sides adjacent to the right angle are 4 inches and 3 inches respectively?

a. 6 sq. inches

b. 7 sq. inches

c. 12 sq. inches

d. 25 sq. inches

e. None of the above

Page 43: Accountability in developing assessment tools

Multiple Choice 8. As compared with the American factory worker in

the early part of the 19th century, the American factory worker at the close of the century

a. was working long hours

b. received greater social security benefits

c. was to receive lower money wages

d. was less likely to belong to a labor union.

e. became less likely to have personal contact with employers

Make options grammatically parallel to each other and consistent with the stem.

Why is the item faulty?

Page 44: Accountability in developing assessment tools

Multiple Choice

IMPROVED: As compared with the American factory worker in the early part of the century, the American factory worker at the close of the century

a. worked longer hours.

b. had more social security.

c. received lower money wages.

d. was less likely to belong to a labor union

e. had less personal contact with his employer

Page 45: Accountability in developing assessment tools

Multiple Choice 9. The “standard error of estimate’

refer to

a. the objectivity of scoring.

b. the percentage of reduced error variance.

c. an absolute amount of possible error.

d. the amount of error in estimating criterion scores.

Avoid such irrelevant cues as “common elements” and “pat verbal associations.”

Why is the item faulty?

Page 46: Accountability in developing assessment tools

Multiple Choice

IMPROVED: The “standard error of estimate” is most directly related to which of the following test characteristic?

a. Objectivity

b. Reliability

c. Validity

d. Usability

e. Specificity

Page 47: Accountability in developing assessment tools

Multiple Choice 10. What name is given to the group of

complex organic compounds that occur in small quantities in natural foods that are essential to normal nutrition?

a. Calorie

b. Minerals

c. Nutrients

d. Vitamins

In testing for understanding of a term or concept, it is generally preferable to present the term in the stem and alternative definitions in the options.

Why is the item faulty?

Page 48: Accountability in developing assessment tools

Multiple Choice

IMPROVED: Which of the following statements is the best description of a vitamin?

Page 49: Accountability in developing assessment tools

Short Answer Type

1. Require short, definite, clear-cut, and explicit answers

FAULTY: Earnest Hemingway wrote______

IMPROVED: The Old Man and the Sea was written by _______.

Who wrote The Old man and the Sea?

Why is the item faulty?

Page 50: Accountability in developing assessment tools

Short Answer Type

2. Avoid multimutilated statements

FAULTY: _____ pointed out in ____ the freedom of thought in America was seriously hampered by ___, ____, & __.

IMPROVED: That freedom of thought in America was seriously hampered by social pressures toward conformity was pointed out in 1830 by ______.

Why is the item faulty?

Page 51: Accountability in developing assessment tools

Short Answer Type

• 3. If several equal answers equal credit should be given to each one.

• 4. Specify and announce in advance whether scoring will take spelling into account.

Page 52: Accountability in developing assessment tools

Short Answer Type

5. In testing for comprehension of terms and knowledge of definition, it is often better to supply the term and require a definition than to provide a definition and require the term.

FAULTY: What is the general measurement term describing the consistency with which items in a test measure the same thing?

IMPROVED: Define “internal consistency reliability.”

Why is the item faulty?

Page 53: Accountability in developing assessment tools

Short Answer Type

6. It is generally recommended that in completion items the blanks come at the end of the statement.

FAULTY: A (an) ________ is the index obtained by dividing a mental age score by chronological age and multiplying by 100.

IMPROVED: The index obtained by dividing a mental age score by chronological age and multiplying by 100 is called a (an) ________

Why is the item faulty?

Page 54: Accountability in developing assessment tools

Short Answer Type

7. Minimize the use of textbook expressions and stereotyped language.

FAULTY: The power to declare war is vested in ______

IMPROVED: Which national legislative body has the authority to declare war?

Why is the item faulty?

Page 55: Accountability in developing assessment tools

Short Answer Type

8. Specify the terms in which the response is to be given.

FAULTY: Where does the Security Council of the United Nations hold its meeting?

IMPROVED: In what city of the United States does the Security Council of the United Nations hold its meeting?

FAULTY: If a circle has 4-inch diameter, its area is_____ IMPROVED: A circle has 4-inch diameter. Its area in

square inches correct to two decimal places, is _____

Why is the item faulty?

Page 56: Accountability in developing assessment tools

Short Answer Type

9. In general, direct questions are preferable to incomplete declarative sentences.

FAULTY: Gold was discovered in California in the year ___

IMPROVED: In what year was gold discovered in California?

Why is the item faulty?

Page 57: Accountability in developing assessment tools

Short Answer Type

10. Avoid extraneous clues to the correct answer

FAULTY: A fraction whose denominator is greater than its numerator is a _____

IMPROVED: Fractions whose denominator are greater than their numerators are called _____

Why is the item faulty?

Page 58: Accountability in developing assessment tools

• Ability to Recognize the Relevance of Information

58

Page 59: Accountability in developing assessment tools

• Ability to Recognize Warranted and Unwarranted Generalizations

59

Page 60: Accountability in developing assessment tools

• Ability to Recognize Inferences

60

Page 61: Accountability in developing assessment tools

• Ability to Interpret Experimental Findings

61

Page 62: Accountability in developing assessment tools

• Ability to Apply Principles

62

Page 63: Accountability in developing assessment tools

• Ability to Recognize Assumptions

63

Page 64: Accountability in developing assessment tools

Reading comprehension • Bem (1975) has argued that androgynous people are

“better off” than their sex-typed counterparts because they are not constrained by rigid sex-role concepts and are freer to respond to a wider variety of situations. Seeking to test this hypothesis, Bem exposed masculine, feminine, and androgynous men and women to situations that called for independence (a masculine attribute) or nurturance (a feminine attribute). The test for masculine independence assessed the subject’s willingness to resist social pressure by refusing to agree with peers who gave bogus judgments when rating cartoons for funniness (for example, several peers might say that a very funny cartoon was hilarious). Nurturance or feminine expressiveness, was measured by observing the behavior of the subject when left alone for ten minutes with a 5-month old baby. The result confirmed Bem’s hypothesis. Both the masculine sex-typed and the androgynous subjects were more independent (less conforming) on the ‘independence” test than feminine sex-typed individuals. Furthermore, both the feminine and the androgynous subjects were more “nurturant” than the masculine sex-typed individuals when interacting with the baby. Thus, the androgynous subjects were quite flexible, they performed as masculine subjects did on the “feminine” task.

64

35. What is the independent variable in the study? a. Situations calling for independence and

nurturance b. Situation to make the sex type react c. Situations to make the androgynous be

flexible d. Situations like sex type, androgynous and

sex role concepts 36. What are the levels of the IV? a. masculine attribute and feminine attribute b. rating cartoons and taking care of a baby c. independence and nurturance d. flexibility and rigidity

Page 65: Accountability in developing assessment tools

Interpreting Diagrams Instruction. Study the following illustrations and answer the following

questions.

Figure 1

65

Pretest Posttest

101. Which group received the treatment? a. group A b. group B b. c. none of the above 102. Why did group B remain stable across the experiment? a. there is an EV b. there is no treatment c. there is the occurence of ceiling effect 103. What is the problem at the start of the experiment? a. the groups are nonequivalent b. the groups are competing with each other c. the treatment took place immediately

Page 66: Accountability in developing assessment tools

Workshop 1

• Work in groups by department and write one course objective and write one item to match the objective.

• Ensure that the items is aligned with the objective • Conditions: Skills should cover undertstanding,

applying, analyzing, evaluation, use multiple choice • Organize the presentation in a table: • Group presentation and feedback will follow.

Course Objective Item

Page 67: Accountability in developing assessment tools

Characteristics of performance-based assessment

• Students perform, create, construct, produce, or do something.

• Deep understanding and/or reasoning skills are needed and assessed.

• Involves sustained work, [often days and weeks].

• Calls on students to explain, justify, and defend.

• Performance is directly observable.

• Involves engaging in ideas of importance and substance.

• Relies on trained assessor’s judgments for scoring

• Multiple criteria and standards are prespecified and public

• There is no single correct answer.

• If authentic, the performance is grounded in real world contexts and constraints.

Page 68: Accountability in developing assessment tools

Learning Targets

• Skills – Communication and presentation skills – Ex: Speaking

1. Speaking clearly, expressively, and audibly a. Using voice expressively b. Speaking articulately and pronouncing words correctly c. Using appropriate vocal volume

2. Presenting ideas with appropriate introduction, development, and conclusion

1. Presenting ideas in an effective order 2. Providing a clear focus on the central idea 3. Providing signal words, internal summaries, and transitions

Page 69: Accountability in developing assessment tools

Learning Targets

3. Developing ideas using appropriate support materials

a) Being clear and using reasoning processes

b) Clarifying, illustrating, exemplifying, and documenting ideas

4. Using nonverbal cues

a. Using eye contact

b. Using appropriate facial expressions, gestures, and body movement

5. Selecting language to a special purpose

a. Using language and conventions appropriate for the audience

Page 70: Accountability in developing assessment tools

Learning Targets

• Psychomotor skills – Fine motor: cutting papers with scissors, drawing a

line tracing, penmanship, coloring drawing, connecting dots

– Gross motor: Walking, jumping, balancing, throwing, skipping, kicking

– Complex: Perform a swing golf, operate a computer, drive a car, operate a microscope

– Visual: Copying, finding letters, finding embedded figures, identifying shapes, discrimination

– Verbal and auditory: identify and discriminate sounds, imitate sounds, pronounce carefully, blend vowels

Page 71: Accountability in developing assessment tools

Learning Targets

• Products

– Write promotional materials

– Report on a foreign country

– Playing a new song

Page 72: Accountability in developing assessment tools

Variation of authenticity

Relatively authentic Somewhat authentic Authentic

Identify the materials used in dressing a wound

Give the steps in dressing a wound

Dresses the wound of a patient

Tell the use of a thermometer

Records temperature in a chart

Uses the thermometer on a patient and records it.

Explain the steps on taking blood pressure

Show how to take the blood pressure using a sphygmomanometer

Get the blood pressure of a patient.

Page 73: Accountability in developing assessment tools

Complexity of task

• Restricted-type task – Narrowly defined and require brief responses – Task is structured and specific – Ex:

• Construct a bar graph from data provided • Demonstrate a shorter conversation in French about what is on a

menu • Read an article from the newspaper and answer questions • Flip a coin ten times. Predict what the next ten flips of the coin will

be, and explain why. • Listen to the evening news on television and explain if you believe

the stories are biased. • Construct a circle, square, and triangle from provided materials

that have the same circumference.

Page 74: Accountability in developing assessment tools

Constructing Performance Based tasks

• Extended-type task – Complex and elaborate – Often include collaborative work with small group of

students. – Requires the use of a variety of information – Examples:

• Design a playhouse and estimate cost of materials and labor • Plan a trip to another country: Include the budget and itinerary,

and justify why you want to visit certain places • Conduct a historical reenactment (e. g. impeachment trial of

ERAP) • Diagnose and repair a car problem • Design an advertising campaign for a new or existing product

Page 75: Accountability in developing assessment tools

Identifying Performance Task Description

• Prepare a task description • Listing of specifications to ensure that essential if

criteria are met • Includes the ff.:

– Content and skill targets to be assessed – Description of student activities

• Group or individual • Help allowed

– Resources needed – Teacher role – Administrative process – Scoring procedures

Page 76: Accountability in developing assessment tools

Performance-based Task Question Prompt

• Task prompts and questions will be based on the task descriptions.

• Clearly identifies the outcomes, outlines what the students are encourage dot do, explains criteria for judgment.

Page 77: Accountability in developing assessment tools

Performance Criteria

• What you look for in student responses to evaluate their progress toward meeting the learning target.

• Dimensions of traits in performance that are used to illustrate understanding, reasoning, and proficiency.

• Start with identifying the most important dimensions of the performance

• What distinguishes an adequate to an inadequate demonstration of the target?

Page 78: Accountability in developing assessment tools

Example of Criteria

• Learning target: – Students will be able to write a persuasive paper to

encourage the reader to accept a specific course of action or point of view.

• Criteria: – Appropriateness of language for the audience – Plausibility and relevance of supporting arguments. – Level of detail presented – Evidence of creative, innovative thinking – Clarity of expression – Organization of ideas

Page 79: Accountability in developing assessment tools

Example of Performance-based task

• Final Product • Chemists are often required to determine the

concentration of unknown acidic or basic solutions. You will be provided with an unknown solution that is either acidic or basic. Your problem is to design an experiment, using the materials (and/or others) listed below, to determine the concentration of the unknown solution provided. Express your concentration of the unknown material in moles of H+ (aq) per liter or moles of OH- (aq) per liter of solution.

Page 80: Accountability in developing assessment tools

Task • a) Under the heading PROCEDURE list in order the steps of the procedure

you will use to solve the problem. You may include a diagram to help illustrate your plans for the experiment. Include any safety procedures you would follow.

• b) Construct a DATA TABLE or indicate any other method that you could use to record the observations and results that will be obtained.

• Materials

• 2 beakers, 100 ml 1 burette, 50 ml 4 Erlenmeyer flasks, 250 ml

• 1 filter funnel, small 1 graduated cylinder, 50 ml 1 burette stand

• safety goggles calculator 1 wash bottle of distilled watre

• 1 white tile hydrochloric acid, 0.1 mol/L sodium hydroxide, 0.1 mol/L

• phenolphthalein indicator bromothymol blue indicator unknown solution

Page 81: Accountability in developing assessment tools

Criteria Yes NO A. Procedures for investigation

1. Detailed

procedure/experimentally

feasible

Method includes details which will clearly

result in accurate determination of

concentration of unknown, including steps

below and repeated trials.

2. Sequence to plan Suggests method to determine molarity of

unknown solution

3. General strategy

First identifies unknown solution as an acid

or base using appropriate indicator, then

suggests titration using neutralizing solution.

4. Safety procedures Safety goggles should be worn; care in

handling acids and bases.

5. Use of equipment/diagram

Appropriate equipment and materials are

suggested, especially use of burette and

indicator.

Page 82: Accountability in developing assessment tools

Criteria Yes No B. Plan for recording and

organizing observations/data

1. Space for manipulation of data

or qualitative description

Could include space for averaging data;

stipulation of indicator used.

2. Matched to plan

Plan is comprehensive so that all

observations and data generated by plan

may be recorded; e.g., space for repeated

trials is included.

3. Organized sequentially Organized so that recording follows data

as generated.

4. Labelled fully (units included) All columns and rows are identified;

correct units of measurement are used.

5. Variables identified Solution used to titrate unknown is

indicated as acid or base.

Page 83: Accountability in developing assessment tools

Rating Scales

• Indicate the degree to which a particular dimension is present.

• Three kinds:

– Numerical

– Qualitative

– Combined Qualitative and Quantitative

Page 84: Accountability in developing assessment tools

Rating Scale

• Numerical Scale

– Numbers of a continuum to indicate different level of proficiency in terms of frequency or quality

Example: Complete Understanding 5 4 3 2 1 No understanding

Clear organization 5 4 3 2 1 No organization

Fluent reader 5 4 3 2 1 Emerging reader

Page 85: Accountability in developing assessment tools

Rating Scale

• Qualitative scale

– Uses verbal descriptions to indicate student performance.

– Provides a way to check the whether each dimension was evidenced.

• Type A: Indicate different gradations of the dimension

• Type B: Checklist

Page 86: Accountability in developing assessment tools

Rating Scale

• Example of Type A: – Minimal, partial, complete – Never, seldom, occasionally, frequently, always – Consistent, sporadically, rarely – None, some, complete – Novice, intermediate, advance, superior – Inadequate, needs improvement, good excellent – Excellent, proficient, needs improvement – Absent, developing, adequate, fully developed – Limited, partial, thorough – Emerging, developing, achieving – Not there yet, shows growth, proficient – Excellent, good, fair, poor

Page 87: Accountability in developing assessment tools

Rating Scale

• Example of Type A: Checklist

Page 88: Accountability in developing assessment tools

Rating Scale

• Holistic scale – The category of the scale contains several criteria, yielding

a single score that gives an overall impression or rating

Example level 4: Sophisticated understanding of text indicated with constructed meaning

level 3: Solid understanding of text indicated with some constructed meaning

level 2: Partial understanding of text indicated with tenuous constructed meaning

level 1: superficial understanding of text with little or no constructed meaning

Page 89: Accountability in developing assessment tools

Example holistic scale

Page 90: Accountability in developing assessment tools

Analytic scale

• Analytic Scale

– One in which each criterion receives a separate score.

Example

Criteria Outstanding 5 4

Competent 3

Marginal 2 1

Creative ideas

Logical organization

Relevance of detail

Variety in words and sentences

Vivid images

Page 91: Accountability in developing assessment tools

Rubrics

• When scoring criteria are combined with a rating scale, a complete scoring guideline is produced or rubric.

• A scoring guide that uses criteria to differentiate between levels of student proficiency.

Page 92: Accountability in developing assessment tools

Example of a rubric

Page 93: Accountability in developing assessment tools

• Rubrics should answer the following questions: – By what criteria should performance be judged?

– Where should we look and what should we look for to judge performance success?

– What does the range in the performance quality look like?

– How do we determine validity, reliability, and fairly what scores should be given and what that score means?

– How should the different levels of quality be described and distinguished from one another?

Rubric

Page 94: Accountability in developing assessment tools

Workshop

• Work in groups by department and create an performance task for one objective.

• Ensure that the performance task is aligned with the objective

• Organize the presentation in using the following: – Objective – Describe the final output – Describe what the students will do – List the criteria

• Group presentation and feedback will follow.