Evaluating Open Educational Resource

(OER) Objects

Rubric IV: Quality of Assessments

CC BY Achieve 2013

2

Open: Containing an open license - no restrictions on remixing or reusing

[Having no enclosing or confining barrier; not restricted to a particular group or category of participants]

Open Educational Resources:[With Webster’s Definitions]

O

E

R

Educational: Used for teaching and learning

[Pertaining to the action or process of educating or being educated]

Resource: Object used to support an effort or task

[A source of support or aid, especially one that can be readily drawn upon when needed]

3

What are OER?

• OER are teaching, learning, and research resources that contain an open license.

• They provide extraordinary opportunities for educators to freely share knowledge and resources.

• They offer great potential for instructional innovation and networks for sharing best practices.

• There are millions of OER objects available online.

Open Educational Resources (OER)

4

OER objects may include (but are not limited to):

• Images

• Applets

• Games

• Worksheets

• Lesson plans

• Original-source texts

• Assessments

• Units

• Textbooks

• Teacher/learner support materials

NOTE: Any smaller component of a more complex object, that can exist as a stand-alone, also qualifies as an “object” (e.g. a unit in a textbook, a lesson in a unit, or an activity in a lesson).

Open Educational Resources (OER)

5

The Achieve OER Evaluation Rubrics

How and why were the Achieve OER Rubrics created?

• Educators using the vast system of sharing need a method for filtering OER to meet the needs of their students.

• The purpose of the rubrics is to provide a structure for evaluating an online resource in a systematic, purposeful and comprehensive way.

• There are two ways of approaching the evaluation of a resource: o Holistically evaluating the object with a single rating, (similar to the star-rating

systems used on Netflix, Yelp and Amazon)o Separately evaluating each of the components of quality, as with the Achieve

OER Rubrics.

• The rubric criteria are based on Achieve protocols used to assist states in alignment and quality review studies.

• The Achieve OER Evaluation Tool, hosting the rubrics on OERCommons.org, allows a user to identify evaluate and sort objects based on the specified essential elements of quality.

6

How do the Achieve OER Rubrics work?

• The rubrics represent an evaluation system for objects found within Open Educational Resources.

• They are hosted as an online evaluation tool on the repository, OERCommons.org, but can also be used independently.

• They are applied to any content area. o NOTE: At this stage only Common Core Standards for English Language Arts/Literacy

and Mathematics are available in the Achieve OER Evaluation Tool. Content standards for History, Social Studies, Sciences, and technical subjects are not yet available.

• Each rubric is applied independently to an object.

• They rate the potential, not the actual, effectiveness of an object.

• They use a five-point scoring system to describe levels of quality.

Applying the Achieve OER Rubrics

7

This presentation will focus on Rubric IV:

Rubric I. Degree of Alignment to Standards

Rubric II. Quality of Explanation of the Subject Matter

Rubric III. Utility of Materials Designed to Support Teaching

Rubric IV. Quality of Assessment

Rubric V. Quality of Technological Interactivity

Rubric VI. Quality of Instructional Tasks and Practice Exercises

Rubric VII. Opportunities for Deeper Learning

o NOTE: Sometimes one or more of the rubrics may not be relevant to a particular object. In those cases, a rating of N/A is appropriate.

The Achieve OER Evaluation Tool Rubrics

8

How and Why is Rubric IV Applied?

• Applied to objects which seek to determine what a student knows.

• Assessments may be used to assess knowledge before, during, or after a topic is taught.

• Applied to entire sets of assessment items when multiple items are included in an object.

• Not applicable for an object not designed to have an assessment component, even when one might imagine aspects of the object that may be used for assessment purposes.

Rubric IV:Quality of Assessments

9

3: An object is rated superior only if all of the following are true:

• All of the skills and knowledge assessed align clearly to the content and performance expectations intended, as stated or implicit in the object.

• Nothing is assessed that is not included in the scope of intended material unless it is differentiated as extension material.

• The most important aspects of the expectations are targeted and are given appropriate weight/attention in the assessment.

• The assessment modes used in the object, such as selected response, long and short constructed response, or group work require the student to demonstrate proficiency in the intended concept/skill.

• The level of difficulty is a result of the complexity of the subject-area content and performance and of the degree of cognitive demand, rather than a result of unrelated issues (e.g. overly complex vocabulary used in math word problems).

For example a superior object for Rubric IV has appropriately focused and weighted assessments included, of varying types and modes, with accurate solution keys provided.

Rubric IV:Quality of Assessments

10

2: An object is rated strong if it assesses all of the content and performance expectations intended, but the assessment modes used do not consistently offer the student opportunities to demonstrate proficiency in the intended concept/skill.

For example an object with an assessment component intended to determine a student’s ability to write a proof or present an argument uses only multiple choice or matching assessment items. This object is considered strong for Rubric IV because it has clear requirements that students know the structure and parts of the presentation of a proof or argument, but lacks an assessment mode that offers students an opportunity to fully demonstrate the intended skill/knowledge: writing a proof or argument.

Rubric IV:Quality of Assessments

11

1: An object is rated limited if it assesses some of the content or performance expectations intended (as stated or implicit in the object), but omits some important content or performance expectations and/or fails to offer the student opportunities to demonstrate proficiency in the intended content/skills.

For example an object’s stated intention is to teach students to write fractions, with denominators of 2, 4, 5, 8, 10, and 100, as decimals, but assessment items address only measurements with the denominators 2, 4 and 8. The object is considered limited in the information and performance expectations it assesses.

Rubric IV:Quality of Assessments

12

0: An object is rated very weak or no value if assessments contain significant errors, do not assess important content/skills, are written in a way that is confusing to students, or are unsound for other reasons.

For example the following are considered very weak for Rubric IV: 1) an object that includes general suggestions for the type of assessments to be used (formative, summative, self-, pre-, etc.) but is missing specific questions or even question types; 2) one that has attached student assessments that address content not included in the lesson; or 3) one that includes some assessment items with a lesson, but with incorrect solutions. o NOTE: This list is not meant to be exhaustive. There are other possible examples

of 0-rated objects for Rubric IV.

Rubric IV:Quality of Assessments

13

N/A: This rubric is not applicable (N/A) for an object that is not designed to have an assessment component.

Even if one might imagine ways an object could be used for assessment purposes, if it is not its intended use, not applicable is the appropriate score.

For example an object is an interactive student-directed game that introduces a new concept. There is no intention by the designer to include assessments so Rubric IV is not applicable to the object.

Rubric IV:Quality of Assessments

Using the Achieve OER Evaluation Tool On OERCommons.org

Rubric IV

o NOTE: Specific instructions for Using the OERCommons Website to search for and evaluate objects, using all of the rubrics, can be found in another slide presentation of this training package.

15

Achieve OER Evaluation Toolon OERCommons.org

OERCommons.org hosts the rubrics as the Achieve OER Evaluation Tool

16

Rubric IV:Quality of AssessmentsAchieve OER Evaluation Tool

Using OERCommons.org to evaluate an object for its quality of assessments:

After you have evaluated the object using Rubrics I, II, and III, you are ready to apply Rubric IV.

Begin by clicking on at the end of Rubric III or on the title line for Rubric IV.

17

To view a video about Rubric IV click here…

Rubric IV:Quality of AssessmentsAchieve OER Evaluation Tool

18

Hover over the rating numbers to see an abbreviated version of the rating descriptors.

Review the object with your focus on the quality of the object’s assessments and select the appropriate rating.

Rubric IV:Quality of AssessmentsAchieve OER Evaluation Tool

19

To download a full text version of the rubrics click here:

Rubric IV:Quality of AssessmentsAchieve OER Evaluation Tool

20

Rubric IV:Quality of AssessmentsAchieve OER Evaluation Tool

Across the bottom of Rubric IV you see options to:

• Leave a [+ Comment],

• Clear rating,

• [Save & Go to the Next Rubric] (You can also click on any rubric title to leave Rubric IV, but your ratings will not be saved.)

21

Now you are ready to rate the object using the Achieve OER Evaluation Tool’s other rubrics. Rubrics can be used in any order and can be selected by simply clicking on the rubric title.

Remember to click after each rubric to ensure that your work is saved.

You can tell if your work was saved by the symbol at the beginning of each rubric title: indicates not yet rated or rating was not saved

indicates your rating has been saved

Achieve OER Evaluation Toolon OERCommons.org: Rubric IV

Slides developed by Achieve:www.achieve.org1400 16th Street, NW / Suite 510 Washington, DC 20036