Upload
buck-institute-for-education
View
3.642
Download
3
Tags:
Embed Size (px)
DESCRIPTION
A federally-funded, randomized controlled trial showing efficacy of PBL use in economics with professional development
Citation preview
Effects of Problem Based Economics on High School
Economics Instruction
Neal D. FinkelsteinRegional Educational Laboratory, West (REL West), WestEd
Jason RavitzBuck Institute for Education
AERA Annual Meeting – New Orleans, LAApril 14, 2011
1
Neal D. FinkelsteinSenior Research Scientist
REL West)WestEd
December 8, 2010
2
The case for economic literacy is obvious. High school graduates will be making economic choices all their lives, as breadwinners and consumers, and as citizens and voters. A wide range of people will be bombarded with economic information and misinformation for their entire lives. They will need some capacity for critical judgment. They will need it whether or not they go to college.
James Tobin, Yale Economist and Nobel Laureate Wall Street Journal, July 9, 1986
3
Purpose• Assess student-level impacts of a problem-
based instructional approach to high school economics.
• Tests the effectiveness of Buck Institute for Education curriculum on student learning of economics content and problem solving skills.
• Within-school randomized controlled trial in urban and rural high schools in California and Arizona during the 2007/08 school year.
4
Why Study Economics Instruction?• For decades, economists, educators, and business and
government leaders have advocated for economic literacy as a component in school curricula.
• In general, high school economics does not help students understand our economic system.
• Most teachers are not adequately prepared to teach economics.
• 48 states include content standards in economics. 40 require implementation of standards, 23 require testing, and 17 require an economics course for graduation.
5
6
• Each of five curriculum modules is built around a case study well-suited to student-driven problem solving.
– Teachers use economic problems to help students learn to contextualize, understand, and solve problems.
• Teaching guide lays out all aspects of instruction
– including problem statement, placement in curriculum, concepts taught, objectives, resources materials and do’s and don’ts.
• Teachers in the intervention group receive professional development introducing the curriculum.
– Led by expert teachers with experience using the PBE units
What is Problem-Based Economics Instruction?
Students are often challenged in high school Economics courses because:
O Economics concepts can be abstract
O They don’t see relevance to real life
O They are bored
O All of the Above
7
“Anyone…? Anyone…?”
8
The Curriculum
Organizes curriculum and instruction around• Carefully crafted situations• Adapted from real-world issues• With more than one possible solution
Students work in teams to gather, apply and demonstrate knowledge and skills as they develop and present solutions.
9
• Units on micro- & macro-economics (7 total, 5 used in study)
• Aligned with National Content Standards in Economics and state standards
• Units take from one to three weeks
• Typically taught in stand-alone high school Economics courses
10
HOW EACH PBE UNIT IS STRUCTURED:
1. Project Launch (do NOT pre-teach!)
2. Framing the Inquiry (Know/Need to Know List & Driving Question)
3. Problem-Solving & Learning Activities
4. Presentation, Assessment, and Debrief
11
WHAT IS INCLUDED IN EACH PBE UNIT:
• guidance on teaching with PBL
• “Make More Money?” activity
• step-by-step directions (with Project Log prompts and Content Teaching Notes)
• student handout masters
• content background material for teacher mini-lectures & lessons
• rubrics & exemplars for assessment
• multiple-choice test
12
PBE UNIT: RUNNING IN PLACE
• As grad students in Economics, students are asked for advice by an entrepreneur who wants to start a new shoe company
• Students hear from consumers and producers
• Students learn about factors of production, circular flow
• Students create visuals and make an oral presentation of their recommendations
13
“Making economics more engaging and relevant for students is possible!”
“I have really enjoyed teaching with this curriculum; having a central problem to refer back to throughout the unit helps with teaching the concept.”
“I just finished using PBE for the second time. The material that my kids are putting together is amazing. These are inner-city kids and the results are beyond belief. They love your material. I have been teaching for over 30 years and the lessons are undoubtedly the best I have ever used.”
WHAT TEACHERS SAY ABOUT PBE:
14
Conceptual Framework
15
• Student achievement outcomes are of primary importance and are hypothesized to be mediated by changes in teacher knowledge and pedagogical practice.
16
Logic model for the study of high school instruction with Problem-based Economics
Teacher Professional Development
Classroom Instruction
Student Performance
Content• Review of economic
theory and related applications; curriculum reivew
Pedagogy• Problem-based
approach to teaching and support strategies for students
Curriculum• Fluent presentation
of curriculum; scaffolding for students
Engagement• Problem-solving
strategies that build economic content and reinforce core analytic thinking
Concepts• Broad-based
literacy in micro- and macroeconomic concepts
Problem Solving• Ability to apply
concepts and analytic approaches to real-world problems
Conceptual Framework
Research Questions1. Does PBE change students’ content knowledge in
economics?
2. Does PBE change students’ problem-solving skills in economics
3. Does PBE change teachers’ content knowledge of economics?
4. Does use of PBE change economics teachers’ instructional practices
5. Does the use of PBE change teachers’ satisfaction with teaching materials and methods used to teach economics?
17
Key Outcomes to be Studied and Their Measures
• The primary outcome measure is content knowledge gains for students in economics measured by the Council for Economic Education’s Test of Economic Literacy.
• Student problem-solving skills are measured with open-response performance assessment of applied economic concepts developed by UCLA CRESST.
• Teachers’ content knowledge was also measured using the Test of Economic Literacy. Pedagogical practices and teacher satisfaction were measured using a series of survey measures.
18
Key Study Characteristics
19
Study design Cluster-randomized trial
Unit of assignment Teachers
Statistical power estimatesFor Type 1 error = .05, 80 percent or higher power to detect minimum detectable effect size of 0.18-0.21 at student level and 0.55 at teacher levela
Implementation began Summer 2007
Student measures
Test of Economic Literacy (pre/post)
Student surveys (pre/post)
Performance task assessments
Administered January 2008, June 2008
Administered January 2008, June 2008
Administered June 2008
Teacher measures
Test of Economic Literacy (pre/post)
Teacher surveys (pre/post)
Administered June–August 2007, June 2008
Administered June–August 2007, June 2008
Note: a. The estimates were based on 83 teachers, with an average of 40 students per teacher.
Study characteristics and data collection schedule
Teacher Participants
20
Number of recruited teachers: n = 128 teachers from 106 schools
Excluded: 0 teachers
Number of teachers randomized: n = 128 teachers from 106 schools
Allocated to intervention group: n = 64• 44 teachers from 44 singleton
schools• 20 teachers from 16 schools
with 2 or more participating teachers.
Allocated to control group: n = 64• 46 teachers from 46 singleton
schools• 18 teachers from 15 schools
with 2 or more participating teachers.
Included: n = 42• 28 teachers from 28 singleton
schools• 14 teachers from 11 schools
with 2 or more participating teachers.
Not Included: n = 22• 16 teachers from 16 singleton
schools• 6 teachers from 6 schools
with 2 or more participating teachers.
Baseline Measures (Summer 2007)• Teacher content knowledge in
economics (n = 41)• Pedagogical practices used (n = 39)• Satisfaction with teaching materials
& methods (n = 40)
Outcome Measures (June 2008)• Teacher content knowedge in
economics (n = 38)• Pedagogical practices used (n = 38)• Satisfaction with materials &
methods (n = 37)
Included: n = 41• 31 teachers from 31 singleton
schools• 10 teachers from 9 schools
with 2 or more participating teachers.
Not Included: n = 23• 15 teachers from 15 singleton
schools• 8 teachers from 8 schools
with 2 or more participating teachers.
Baseline Measures (Summer 2007)• Teacher content knowledge in
economics (n = 29)• Pedagogical practices used (n = 25)• Satisfaction with teaching materials
& methods (n = 29)
Outcome Measures (June 2008)• Teacher content knowedge in
economics (n = 34)• Pedagogical practices used (n = 35)• Satisfaction with materials &
methods (n = 35)
Allocation Retention Data Collected & Analyzed
Teacher Participants Providing Student-Level Data
21
Number of teachers recruited and randomized: n = 128 teachers
Allocated to intervention group: n = 64
Attrition: n = 22
Allocated to control group: n = 64
Attrition: n = 23
Number of teachers remained in the study: n = 42
In fall 2007:• 38 teachers returned student-level data• 3 teachers did not return any student-level
data
In spring 2008:• 35 teachers returned student-level data• 6 teachers (including those who dropped
out of the study) did not return any student-level data
Number of teachers remained in the study: n = 41
In fall 2007:• 38 teachers returned student-level data• 3 teachers did not return any student-level
data
In spring 2008:• 29 teachers returned student-level data• 12 teachers (including those who dropped
out of the study) did not return any student-level data
Total number of teachers who provided student-level data in spring 2008: n = 64 teachers
Student Participants
22
Number of students consented to participate in the study & their data available to the research team: n = 4,350 students from 64 teachers
In intervention group (35 teachers):
• Included: n = 2,502• Not Included: n =0
In control group (29 teachers):
• Included: n = 1,848• Not Included: n = 0
Baseline Measures (January 2008)• Student content knowledge in
economicsNon-missing: n = 2,232 (98%)Missing: n = 270 (11%)Number of teachers = 35
Outcome Measures (June 2008)• Student content knowledge in
economicsIncluded: n = 2,178 (87%)Not Included: n = 324 (13%)Number of teachers = 35
• Student problem-solving skillsIncluded: n = 1,918 (77%)Not Included: n = 584 (23%)Number of teachers = 33
Baseline Measures (January 2008)• Student content knowledge in
economicsNon-missing: n = 1,589 (86%)Missing: n = 259(14%)Number of teachers = 29
Outcome Measures (June 2008)• Student content knowledge in
economicsIncluded: n = 1,574 (85%)Not Included: n = 274 (15%)Number of teachers = 29
• Student problem-solving skillsIncluded: n = 1,497 (81%)Not Included: n = 351 (19%)Number of teachers = 29
Experimental Condition Data Collected & Analyzed
Data Analysis Methods
23
• The analyses for this study compare outcomes for students and teachers in the treatment group with their counterparts in the control group after the economics course has been completed.
• For student outcomes, the analyses involve fitting conditional multilevel regression models (HLM) to account for the nesting of students within teachers.
Data Analysis Methods
24
• A random effect for teachers is included in the model to account for the nesting of student observations within teachers. Fixed effects include treatment status, baseline (pre-test) measures of outcome variables, and other student and teacher-level covariates.
• For teacher outcomes, single-level regression models that include treatment status along with certain covariates (such as pre-test measures of outcome variables and teacher background information) are used to estimate program impacts.
25
• Statistically significant finding: students whose teachers had received professional development and support in PBE outscored their control group peers on the TEL by an average of 2.6 test items.
Student-Level Findings
• The outcomes on student measures of problem-solving skills and application to real world economic dilemmas also showed significant differences in favor of the intervention group.
Student-Level Findings
26
27
Teacher-Level Findings
• No statistically significant difference between the interventions and control groups on teachers’ knowledge of economics.
• No statistically significant difference in teachers’ pedagogical style with the survey measures used.
• Statistically significant differences in favor of the intervention group teachers on a measure of satisfaction with the teaching materials and methods.
Effects of Problem Based Economics on High School
Economics Instruction
http://ies.ed.gov/ncee/edlabs/regions/west/pdf/REL_20104012.pdf
28