Dennis Sunal, Cynthia Sunal, Cheryl Sundberg, Glenda Ogletree, & Erika SteeleThe University of Alabama

Cheryl L. Mason & Corrine LardySan Diego State University

Dean Zollman & Mojgan Matloob-Haghanikar Kansas State University

College Science Teachers (SCST)

March 27–30, 2008, Boston, MA

TPC 0554594

Impact of Undergraduate Science Course Reform on Student Outcomes

NSEUS

Collaborative Partners: University of Alabama San Diego State University Kansas State University

National Study of Education in Undergraduate Science

Overview

1. Background

• Summary of previous research & problem

2. NSEUS Project Plan

• NSEUS goals• What is a

reformed course?

• Research questions

• Participants3. NSEUS Results to

date• Data collection &

analyses• Pilot Study• National Survey• Timeline• Implications

This study addresses the overall problem

How do undergraduate entry-level science courses, differing in level of reform, affect student learning outcomes?

Study components described1) Literature review of research on previous

undergraduate science reform,

2) National survey of faculty and courses at institutions involved in a professional development program aimed at undergraduate reform program,

3) Pilot study determining the feasibility or proof of concept, of the procedures and instruments for gathering data, and

4) Description of a national study of a population of institutions involved in reforming undergraduate courses with the intention of improving student outcomes.

1. Literature review of previous undergraduate science

reform: Summary of Findings from the

Literature Reform courses use inquiry-based

instruction to increase student achievement

Process of reform positively impacts use of reform pedagogy in undergraduate science

Graduates of reform courses more likely to use inquiry science teaching

Summary (cont.)

Patterns emerging from the studies indicate reform coursework is possible, has positive effects on participants, and must overcome barriers.

Patterns suggest strategies for planning and implementing reform coursework.

The research base is limited but starting points are suggested in it.

What is a “Reformed” Course?

Incorporates the following characteristics;

• Based on national science standards• Student-centered, i.e. active student

roles• Inquiry-based pedagogy• Builds on students’ prior knowledge• Incorporates interdisciplinary

learning and collaborative approaches

Problem Developed from Literature Question

What is the impact of engaging higher education faculty in implementing standards-based reform in undergraduate science courses on student learning outcomes?

Study will investigate• Short- & long term

effects on learning outcomes

• Comparisons between reform and non-reform courses

• Comparisons of courses demonstrating differing levels of reform

Study Design

NOVA ProgramNASA Opportunities for Visionary Academics Created to develop and disseminate a national

framework for enhancing education of pre-service teachers in science, mathematics, and technology. (103 institutions in NOVA Network)

Mission: to improve the STEM literacy of future teachers by implementing standards- and research-based change nationally in higher education using NASA’s unique content.

NOVA Program Web Site

http://www.nasa.gov/audience/foreducators/postsecondary/grants/NOVA.html

Nova Professional Development Model

Professional Development

Ongoing Research

Collaboration among

institutions

Instructional Technology

Inquiry-based Instruction

Interdisciplinary Collaboration

Standards-based Courses

NASA-based Content

2. NSEUS national survey to characterize study population

Conducted a survey of faculty and courses at institutions involved in a national professional development program

Surveyed faculty and courses at institutions involved in undergraduate course reform

NSEUS Project Timeline Year 1 Revise and

complete literature review

Survey & characterize population

Plan data collection protocols & instruments, select study sample

Conduct training on instruments and in research site protocols for all data collectors, and

Begin pilot study,

NSEUS Study Population

NSEUS National Survey

MA-II5%

BA-GEN14%

DR-EXT13%

DR-INT11%

MA-I48%

BA-LA7%

BA-SPECI1%

AA1%

Description of Institutions (Study Population)

103 institutions 354 faculty in

collaborative teams of 3+

185 courses Diverse

institutions

Institutions Offering Reform Courses

Created under the NOVA ProgramInstitutions Number of Courses

Institutions Offering NOVA-Like Courses

Reform Courses still offered

72 146 41(104)

Reform courses stopped

31 39 8(14)

Total 103 185 49(118)

Instructional Methods Used in Reform Courses

Instructional Method Average % of Time per Week

Lecture 14%

Discussion/Interaction of student groups

15%

Integrated inquiry lab 64%

Integrated use of technology

07%

Selected Reform Course Characteristics

Characteristic Average

Years course offered at institution 6

Times offered in a year 2

Course enrollment 35

Minority enrollment 25%

Credit hours 4 (range = 3 - 5)

Sections per semester 1.5 (range = 1 - 6)

Students enrolling in reform courses per year

~10,000

3. NSEUS case study to determine the feasibility of the

procedures and instruments

Case (pilot) study to determine validity and reliability of

• Instrumentation• Data collection

protocols• Site visit

protocol Classroom science learning context (climate)

NSEUS Project Timeline Year 2 Select study

sample Conduct pilot

(case) studies with on site data collection using 2 institutions, 54 undergraduate courses, and 12 elementary teachers

Complete pilot studies

Revise data collection protocols & instruments,

Conduct additional training in research protocols for all data collectors, and

Select 30 study sample institutions from population

Instrumentation (partial list)

Thinking About Science Survey Instrument (TSSI) (Cobern, 2000)

Classroom Learning Environment Survey (CLES) (Taylor & Fraser, 1991, 1997)

Draw-A-Scientist Test (DASTT-C) (Thomas, Pedersen, & Finson, 2001)

Science Teaching Efficacy and Beliefs Instrument (STEBI A & B) (Riggs & Enochs, 1990),

Reformed Teaching Observation Protocol (RTOP) (Sawada, Turley, Falconer, Benford & Bloom, 2002)

PCK Content Representation (CoRe)

&Pedagogical and Professional experience Repertoires (PaP-ers) (Loughran, Mulhall & Berry, 2004).

DAST Results Short-term

misconceptions of science, pre-post with undergraduate courses (lower score represents higher level)

2 institutions, 4 courses (2 reform/ 2 comparison)

Significant change & higher posttest DAST scores

0

10

20

30

40

50

60

70

80

Pre Post

Reformed

Comparison

CLES Results Reform course

students report more positive classroom learning environment (CLES)

Environment related to science learning outcomes

0

10

20

30

40

50

60

70

80

Pre Post

Reform

Comparison

Undergraduate Science Classroom Observations Using the RTOP

Classroom Observations Made in the Reform Undergraduate Science Course Classrooms

Common Classroom Observations Made in Both Types of Undergraduate Science Classrooms

Classroom Observations Made in the Comparison Undergraduate Science Course Classrooms

Extensive student-student interaction during the class

Extensive teacher-student interaction during the class

Lectures were short and provided in a “just in time manner” coordinated with students’ inquiry activities

Lecture and laboratory were integrated

Teachers used technology: smart boards, PowerPoint etc.

Content presented in both courses was current, appropriate, and accurate.

Little requested, or planned, student-student interaction

Teacher lecture took up the majority of the time

Students appeared bored and unengaged with the teacher and the content

Lecture and laboratory were separated in time

Reform Undergraduate Course

Pedagogy in Study Reform Courses – Weekly Time Use

Pedagogy in Study Comparison Courses – Weekly Time Use

Comparison Undergraduate Course

4. NSEUS National StudyNational Study of

a population of institutions involved in a long term professional development program, reforming undergraduate courses with the intention of improving student outcomes.

RESEARCH QUESTIONS

1. What are the essential characteristics of an entry level reformed undergraduate science course?

2. How do reform science course characteristics differ from traditional courses?

3. How do course characteristics relate to undergraduate students short-term learning outcomes?

4. How do characteristics differ between courses with varying degrees of reform?

5. How do varying degrees of reform relate to undergraduate students short-term learning outcomes?

6. How do reform and traditional courses differ in their long-term impacts on K-6 teachers in their own science classrooms?

NSEUS Project Timeline Year 3 Collect & begin

analysis of data from national sample in institutions and elementary classrooms

Conduct data analyses

Year 4 Collect data

from national sample of institutions and elementary classrooms

Conduct analyses of data

Research Participants

30 institutions nationally (stratified random sampling of NOVA institutions), 2 courses from each institution (reformed & comparison)

3 Regions coordinated by 3 Co-PIs: East (UA), Central (KSU), West (SDSU)

Faculty (60), students (~4000), and in-service K-6 teachers (180) of reformed & comparison undergraduate science courses at sample institutions

NSEUS National Study SampleDescription of

Institutions (Study Sample)

30 institutions 142 faculty in

collaborative teams of 3+

60 courses Diverse

institutions

MA-I62%

MA-II3%

BA-GEN6%

BA-LA3%

DR-EXT13%

DR-INT13%

Data Collection Faculty, students, and in-service

teachers Content analysis of course materials Multiple site visits with classroom

observations Student perceptions of science,

science achievement outcome product measures

Interviews and focus groups

NSEUS Project Timeline Year 5 Complete data

collection Complete data

analysis National

dissemination of results and conclusion on undergraduate science reforms

Conduct National Conference on Undergraduate Science Reform and its impact on faculty, undergraduate students, and inservice teachers

Contact Information

http://nseus.org

National Study of Education in Undergraduate Science

TPC 0554594