Copyright © 2004 Educational Testing Service Listening. Learning. Leading. David G. Payne, PhD Senior Executive Director, Client Relations Higher Education

Copyright © 2004 Educational Testing Service

Listening. Learning. Leading.

David G. Payne, PhDSenior Executive Director, Client Relations

Higher Education Division

Patricia ConnorDirector of Higher Education Initiatives

Educational Testing Service

SUNY ConferenceSyracuse, NY

April 27-28, 2005

SUNY General Education Learning Outcomes:Strengthened Campus-based Assessment



2

Overview

ETS Who we are What we bring to the table

Academic Profile and SUNY General Education Learning Outcomes Assessment

Customized SUNY General Education Learning Outcomes Assessment Modules

Key Features of the Customized SUNY Assessments



3

Educational Testing Service

Founded in 1947 by American Council on Education Carnegie Foundation for the Advancement of Teaching College Entrance Examination Board.

Today, as the world’s premier educational measurement and research organization, ETS develops and annually administers more than 11 million tests.

As a Not for Profit organization, ETS has a specific Mission.



4

The ETS Mission

To help advance quality and equity in education by providing fair and valid assessments, research and related services.

Our products and services measure knowledge and skills, promote learning and performance, and support education and professional development for all people worldwide.



5

ETS’s Largest Exams and Annual Volumes

TOEFL – 780,000 GRE – 500,000 GMAT – 229,000 SAT I – 3.1 million AP – 1.4 million PRAXIS – 770,000



6

What else does ETS Provide?

Products and services to support Institutional and Curriculum Development for Higher Education Academic Profile: General Education Test Major Field Tests: Outcomes Assessments in 16

fields of study ICT Literacy: Information and Communication

Technology Assessment Criterion: On-line Writing Evaluation

Free or Low-cost Test Preparation Materials for all Examinees Mission-related services



Academic Profile and

SUNY Learning Outcomes



8

Academic Profile

Test of general education that measures reading, writing, math and critical thinking in the context of the humanities, social sciences and natural sciences.



9

Academic Profile Skill Scores (Norm-Referenced Scaled Scores)

4 Skill Areas in Standard Form Critical Thinking Reading Writing Mathematics

Optional 40 minute Essay



10

Academic Profile:Standard Form

Covers all four skill areas in one test Optional Essay

108 multiple choice questions120 minutesGenerates total and content area scores

for both individuals and groupsProvides demographic dataAllows for locally-written questions



11

Academic Profile and SUNY Learning Outcomes

Mathematics 5 specific skill areas

Basic Written Communication 3 specific skill areas

Critical Thinking 2 specific skill areas



12

Mapping Academic Profile to SUNY Learning Outcomes

Academic Profile Standard Form

•108 questions•Reading•Writing•Mathematics•Critical Thinking

Academic Profile Optional Essay

•40 minutes•Range of Topics•Online Delivery•Holistic Scoring

SUNY General Education Assessment Modules

•40 minutes•36 Questions

OR•40 minutes Essay

•Paper and Pencil or Online Delivery

•Flexible Score Reporting



13

Academic Profile and SUNY Learning Outcomes

Mathematics4 specific skill areas

Basic Written Communication3 specific skill areas

Critical Thinking2 specific skill areas



14

Academic Profile for SUNY General Education

Learning Outcomes Assessment

Tests Available January 2006Report Formats

Paper Electronic

Normative data National Norms SUNY Campuses (depending on volumes)



15

Academic Profile for SUNY

Advantages In use at many

institutions nationwide

Nationally normed Valid Reliable Paper and on-line

versions Web-based score

reporting Group and subgroup

reports

Disadvantages Not precisely

mapped to SUNY Learning Outcomes



Customized SUNY Learning Outcomes

Assessment



17

Four Guiding Principles for Customized SUNY Assessments

1. Customized solution to match SUNY General Education Learning Outcomes

Subscores for as many skill/competencies as possible

2. Ease and efficiency of administration and reporting

Students Faculty Administrators

3. Simple, clear score reports that mirror the SUNY outcomes

4. ETS quality and experience applied to the SUNY needs



18

SUNY Mathematics Module: Skills Assessed

Interpret and draw inferences from mathematical models such as formulas, graphs, tables and schematics

Represent mathematical information symbolically, visually, numerically and verbally

Use arithmetical, algebraic, geometric and statistical methods to solve problems

Estimate and check mathematical results for reasonableness, determine alternatives and select optimal results

Recognize the limits of mathematical and statistical methods



19

Customized SUNY Mathematics Module:

Key Features

Careful Mapping of Skills to Assessment

Format 36 Multiple Choice Items Paper and Pencil Format 40 Minutes

Scores Reported Individual Student Overall Scores 4 Subscores



20

Sample Mathematics Question

Pat purchased 4 large picture frames for $20.00 each, 100 medium picture frames for $10.00 each, and 8 small picture frames for $5.00 each. What was the average (arithmetic mean) price per picture per frame?

A. $7.50

B. 10.00*

C. 11.67

D. 15.00



21

Hotel 1 Person 2 Persons 3 Persons 4 PersonsA $150 $165 $185 $205B 146 166 186 206S 100 115 130 145D 162 177 192 207E 151 151 161 161F 100 110 120 130G 179 179 179 179H 125 135 155 155

According to the information in the table, what is the range of the cost per night for a room for 1 person?

A. $25B. $50C. $62D. $79*

Sample Mathematics Question



22

Customized SUNY Basic Written Communication Module:

Skills Assessed

Produce coherent texts within common college-level written forms

Demonstrate the ability to revise and improve such texts

Research a topic, develop an argument, and organize supporting details



23


Key Features

Careful Mapping of Skills to Assessment Tasks and Format

Short essay (Produce Text) Extended revising and editing of written work tasks

Paragraphs and Short Essays Students will do corrections and make revisions

Paper and Pencil Format 45-50 Minutes

Scores Reported Individual Student Overall Scores 2 Subscores

Text Production Revisions and Corrections



24

Sample Short Essay Task

Some people have suggested that everyone between the ages of 18 and 21 should be required to perform one year of community or government service, such as in the Peace Corps, Environmental Conservancy Corps, a hospital, the military, a rural or inner-city school, or other community outreach projects.

Should government or community service be required of 18-21 year olds?

Support your point of view with reasons from your own experience, observation, or reading.



25

Sample Revision Question

“Being a female jockey, she was often interviewed.”

Rewrite, beginning with She was often interviewed

The next words will beA. “on account of she was”B. “because she was”* C. “by her being”D. “being as she was”



26

Customized SUNY Critical Thinking Module:

Skills Assessed

Critical thinking and analytical writing skills, including the ability to Identify, analyze, and evaluate arguments as

they occur in their own or other’s work; and Develop well-reasoned arguments



27


Key Features

Careful Mapping of Skills to AssessmentTasks and Format

2 Short Essays (20 - 25 minutes each)“Develop an Argument” “Evaluate an Argument”

Task CharacteristicsPaper and pencil formatNo specialized knowledge required



28

Sample Develop Argument Task

Directions: Present your perspective on the issue below, using relevant reasons and/or examples to support your views.

The media (books, film, music, television, for example) tend to create rather than reflect the values of society.



29

Sample Evaluate Argument Task

Directions: Discuss how well reasoned you find this argument.

Hospital statistics regarding people who go to the emergency room after roller-skating accidents indicate the need for more protective equipment. Within this group of people, 75 percent of those who had accidents in streets or parking lots were not wearing any protective clothing (helmets, knee pads, etc.) or any light-reflecting material (clip-on lights, glow-in-the-dark wrist pads, etc.). Clearly, these statistics indicate that by investing in high-quality protective gear and reflective equipment, roller skaters will greatly reduce their risk of being severely injured in an accident.



30


Key Features (continued)

Scoring Essays scored on holistic basis

Scoring can be either human or computer (to be determined based on input from SUNY)

Scores Reported Individual overall score (based on both essays) 2 Subscores

Develop an ArgumentEvaluate an Argument



Score Reporting



32

Sample Individual Student Score Distribution

SUNY Critical Thinking Assessment

0

10

20

30

40

50

60

70

100-103

104-106

107-109

110-112

113-115

116-118

119-121

122-124

125-127

128-130

Scaled score

Num

ber

of s

tude

nts



33

Sample Individual Student Score Distribution

SUNY Mathematics Assessement

0

10

20

30

40

50

60

70

100-103

104-106

107-109

110-112

113-115

116-118

119-121

122-124

125-127

128-130

Scale score

Nu

mb

er

of stu

de

nts



34

Sample MathematicsSubscore Distribution

95

100

105

110

115

120

125S

cale

d S

co

re

Upper Limit

Mean Score

Lower Limit



35

Customized SUNY Mathematics Module:

Subscores

Interpret and draw inferences from mathematical models such as formulas, graphs, tables and schematics

Represent mathematical information symbolically, visually, numerically and verbally

Use arithmetical, algebraic, geometric and statistical methods to solve problems

Estimate and check mathematical results for reasonableness, determine alternatives and select optimal results

x Recognize the limits of mathematical and statistical methods



36


Skills Assessed

Produce coherent texts within common college-level written forms

Demonstrate the ability to revise and improve such texts

Research a topic, develop an argument, and organize supporting details



37


Skills Assessed

Assesses critical thinking and analytical writing skills, including the ability toIdentify, analyze, and evaluate arguments as

they occur in their own or other’s workDevelop well-reasoned arguments



Summaryand

Supplemental Information



39

Customized SUNY General Education


3 Customized Assessments Mapped to SUNY Learning Outcomes Each module is independent Overall Scores and Subscores Easily administered within a class period Paper and Pencil Format Demographic data options Platform that can support future addition of new

modules Option for locally developed questions (to be

determined with input from SUNY Central Administration and and individual campuses)



40



SUNY Outcome Measures will add value beyond Board of Trustees Mandate Accreditation (e.g., MSA) Curriculum development Benchmarking and Quality Improvement Assist in the development of a culture of evidence-

based decision making through campus-based assessment

Respond proactively to public demand for increased accountability in Higher Education



41



Reporting Timetable: 10-20 work days Report Formats

Paper Electronic Data reported directly to campus Individually reliable scores AND subscores Normative data

SUNY Campuses (dependent upon volume) National Comparisons in the future

Demographic breakdowns available (e.g., Major, Year in School)



42

SUNY General Education Learning Outcomes Assessment Modules

ETS can also provide assistance with Sampling plans Data interpretation Assisting campuses in linking results with

curriculum Support for motivating students in learning

outcomes assessments Development of additional assessments to

meet individual campus needsTest Development WorkshopsDevelopment of Standards Based Assessments



43

We welcome feedback!

• Questions/Comments from audienceOR

• E-mail David Payne at:[email protected]

• E-mail Trish Connor at:[email protected]

Documents

Copyright © 2004 Educational Testing Service Listening. Learning. Leading. David G. Payne, PhD Senior Executive Director, Client Relations Higher Education