Upload
anne-kraybill
View
25
Download
0
Embed Size (px)
Citation preview
A R A N D O M A S S I G N M E N T E V A L U A T I O N O F S C H O O L V I S I T S T O A N A R T M U S E U M
J A Y P . G R E E N EB R I A N K I S I D A
D A N B O W E N
U N I V E R S I T Y O F A R K A N S A S
Learning from Crystal Bridges
Motivations for Research
Culturally enriching field trips have a long tradition in public education, but are recently in decline because of budgetary constraints
focus on core subject instruction to maximize performance on high stakes exams
shift to “reward” field trips
What is lost when enriching field trips are cut?
Can we expand the measures of educational outcomes to capture the role of the arts and cultural institutions?
Study Overview
The opening of Crystal Bridges presented a one of a kind opportunity to learn about the effects of school visits to an art museum:
Most area students had never been to an art museum
Heavy demand made random assignment possible
Random assignment is the “gold-standard” in evaluation research; clearly isolates causal impact
Study Overview
Crystal Bridges is the first major art museum built in four decades
Over $800 million endowment, including resources for school tours
In total, 525 K-12 school groups with 38,347 students applied
Applicants were organized into matched pairs
Pairs randomly assigned to treatment and control
Pairs randomly ranked to fill available spots
95 applicants awarded tours (treatment group) and 95 applicants
awarded deferred tours (control group)
Data collected from 174 applicant groups (80 matched pairs) on
average 3 weeks after tour
Enormous Effort
174 applicant groups
123 schools
11,973 miles driven
10,912 students surveyed
489 teachers surveyed
531,781 bits of data
Study Overview: Student Sample
Grade Treatment N Control N Total N
Kindergarten 305 18 323
1st grade 506 634 1,140
2nd grade 624 536 1,160
3rd grade 363 917 1,280
4th grade 924 533 1,457
5th grade 656 1106 1,762
6th grade 635 723 1,358
7th grade 470 243 713
8th grade 204 298 502
9th grade 96 108 204
10th grade 116 276 392
11th grade 147 171 318
12th grade 148 155 303
Total 5,194 5,718 10,912
Study Overview: Student Sample
Student Characteristics Treatment Control Total
Females (K-12 sample) 50.6 50.8 50.7
Ethnicity (3-12 sample)
Hispanic 18.8 18.6 18.7
White 59.3 60.2 59.8
Native American 6.6 5.3 5.8
Black 2.8 3.8 3.3
Marshallese 1.8 2.5 2.2
Asian or Pacific Islander 3.3 2.8 3.0
Mixed race 5.1 4.6 4.8
Other/Unknown 2.3 2.3 2.3
Data Analysis
Estimate of mean differences using the following equation for outcome Y of student i in matched pair m
Yim = α + β1Treati + β2Matchim + εim
This basic regression model is improved by adding controls for observable characteristics
Yim = α + β1Treati + β2Matchim + β3Femalei + β4Gradei + εim
Subgroup effects are estimated with indicator variables and an interaction term where S indicates that a student is a member of a particular subgroup:
Yim = α + β1Treati + β2Matchim + β3Femalei + β4Gradei + β5Si + β6 Si*Treati + εim
Summary of Findings
Students who won the lottery and received a school tour of Crystal Bridges are significantly more likely to:
Recall school visit themes
Demonstrate stronger critical thinking skills
Display higher tolerance
Exhibit greater historical empathy
Develop a taste for art museums and cultural institutions
Benefits are generally significantly larger for disadvantaged students (minority, low-income, or rural schools)
81.7%***
90.5%***
59.9%
66.6%
0% 50% 100%
What does this painting show?
What is the name of this painting?
Control Treatment
What is the name of this painting?
Recalling Themes
Painting C is showing… the importance of women entering the workforce during World War II.
80.5%***
79.2%***
49.4%
27.0%
0% 20% 40% 60% 80% 100%
What does this painting s
plowing it under
Control Treatment
Painting C is showing…An African-American perspective of real and idealized visions of the American dream?
Recalling Themes
Why do you think this man is plowing under his crop rather than harvesting it?
87.9%***
94.1%***
30.0%
45.4%
0% 20% 40% 60% 80% 100%
johnson
Control Treatment
Painting B is showing… abolitionists making maple syrup to undermine the sugar made from slave labor.
Recalling Themes
Painting A is showing… a community in transition to a more industrial lifestyle, from wood-burning to coal-burning.
Thinking Critically about Art
What do you think is going on in this painting?
What do you see that makes you think that?
Scored on 7 Item Rubric
Observing
Interpreting
Evaluating
Associating
Problem Finding
Comparing
Flexible Thinking
Thinking Critically About Art
Group Effect Size Group Effect Size
Full Sample +9.1%**
Smaller Towns +33.4%*** Larger Towns +2.5%
> 40 Minutes +33.6%*** < 40 Minutes +1.0%
Higher Poverty +17.9%*** Lower Poverty -1.5%
First Visit +16.6%***
Tolerance
Survey items included in scale
People who disagree with my point of view bother me.
Artists whose work is critical of America should not be allowed to have their work shown in art museums.
I appreciate hearing views different from my own.
I think people can have different opinions about the same thing.
Tolerance
Group Effect Size Group Effect Size
Full Sample +6.8%**
Smaller Towns +13.3%*** Larger Towns +2.8%
> 40 Minutes +16.8%*** < 40 Minutes +2.6%
Higher Poverty +9.2%** Lower Poverty +0.8%
First Visit +4.0%
“Artists whose work is critical of America should not be allowed to have their work shown in art museums.”
Group Treatment Control Difference
Full Sample 31.9% 34.7% -2.8%***
Smaller Towns 30.2% 33.9% -3.6%*
Larger Towns 33.0% 35.2% -2.2%*
> 40 Minutes 28.1% 34.4% -6.3%**
< 40 Minutes 33.7% 34.9% -1.2%
Higher Poverty 31.6% 37.0% -5.3%***
Lower Poverty 33.2% 29.8% +3.3%***
First Visit 34.5% 35.8% -1.4%
Historical Empathy
Survey items included in scale
I have a good understanding of how early Americans thought and felt.
I can imagine what life was like for people 100 years ago.
When looking at a painting that shows people, I try to imagine what those people are thinking.
Historical Empathy
Group Effect Size Group Effect Size
Full Sample +5.9%**
Smaller Towns +15.0%*** Larger Towns +0.2%
> 40 Minutes +13.8%*** < 40 Minutes +0.6%
Higher Poverty +4.6% Lower Poverty +7.6%
First Visit +12.2%***
“I have a good understanding of how early Americans thought and felt.”
Group Treatment Control Difference
Full Sample 69.7% 65.5% +4.1%***
Smaller Towns 68.8% 61.8% +7.0%***
Larger Towns 70.3% 68.0% +2.3%*
> 40 Minutes 71.0% 63.7% +7.3%***
< 40 Minutes 69.6% 67.1% +2.6%
Higher Poverty 69.7% 66.5% +3.1%**
Lower Poverty 69.7% 63.5% +6.3%*
First Visit 69.3% 63.4% +5.9%***
Cultivating a Taste for Art Museums
Survey items included in scale
I plan to visit art museums when I am an adult.
I would tell my friends they should visit an art museum.
Trips to art museums are interesting.
Trips to art museums are fun.
Would your friend like to go to an art museum on a field trip?
Would you like more museums in your community?
How interested are you in visiting art museums?
If your friends or family wanted to go to an art museum, how interested would you be in going?
Cultivating a Taste for Art Museums
Group Effect Size Group Effect Size
Full Sample +8.0%**
Smaller Towns +22.4%*** Larger Towns +1.6%
> 40 Minutes +20.7%*** < 40 Minutes +2.7%
Higher Poverty +11.0%*** Lower Poverty +4.0%
First Visit +16.8%***
“I would tell my friends they should visit an art museum.”
Group Treatment Control Difference
Full Sample 69.8% 65.7% +4.1%***
Smaller towns 72.5% 62.6% +9.9%***
Larger towns 68.5% 67.1% +1.5%
> 40 Minutes 74.3% 65.1% +9.2%***
< 40 Minutes 67.9% 66.1% +1.9%
Higher Poverty 73.7% 67.6% +6.1%***
Lower Poverty 63.4% 62.2% +1.1%
First Visit 70.1% 63.5% +6.6%***
Behavioral Measure
Treatment Observed Rate
Treatment Expected Rate
Treatment Effect
Total People 57.8% 48.8% +9.0%***
Adults 58.0% 48.8% +9.2%***
Students 57.4% 48.8% +8.6%**
Policy Implications
Visits to cultural institutions have significant benefits for students.
Policymakers need to ensure that schools have resources to take their students on tours of cultural institutions.
School administrators need to decide to use their resources and time for these tours.
Philanthropists need to build and maintain these cultural institutions with quality educational programs.
We need more rigorous research on the effects of culturally enriching experiences on students.
Other Projects
Walton Arts Center
Evaluation of cumulative effect of field trips on students
Museum of Discovery
Measuring effect of field trips on student desire to consume and produce science as well as their factual knowledge about science
Crystal Bridges 2.0
Examination of “spill-over” benefits of school tours on community attendance at museum
Long term evaluation of virtual education program targeted toward rural students