Upload
julianna-shepherd
View
222
Download
1
Tags:
Embed Size (px)
Citation preview
1
Key SINet Impacts:Immediate Impacts
Predictors of Success2-Year Sustained Impacts
7-Year Longevity
Prepared by: Steven H. Shaha, PhD, DBA
Prof. Steven H. Shaha, PhD, DBA
Professor, Center for Public Policy & AdministrationIndependent Evaluator
100+ peer-reviewed publications, 200+ peer-reviewed presentations, 2 books
Advisory and consulting work for 11 governments on four continents
Educational work in 16 States
Professor or Lectured or at 14 universities and graduate schools e.g. UCLA, Harvard, Cambridge (UK), King’s College, London University, South Manchester, Yale, Princeton, Columbia, Cornell, Zayed Univ (UAE)
Advisory and consulting to 50+ non-education-focused organizations, for example:Disney, Ritz-Carlton, Coca-Cola, New Line Cinema, IBM, AT&T, Time Warner, Johnson & Johnson, Marriott, 3 Depts. of Defense, and Pharma: Sanofi, Aventis, Novartis
Education: PhD, Research Methods & Applied StatisticsDBA, Business Administration (PhD)MA, MEd, BS
Disclosures & Bio
4
Executive Summary
• Next Study – Aligned Seminars• Next Study – Aligned
Implementation• Next Study – Aligned Observation
4. Advantages with Longevity into 6th and 7th Years3. Advantages Sustained into Second Year of PD 3602. Prdictors of Success
High vs. Lower Engagement1. Advantage with PD 360
versus none, Year-over-Year
Immediate ImpactsIntent: Define year-to-year changes associated with PD 360Participation
– Systematic sample of 169 elementary schools, in 73 districts, in 19 States– Schools eligible for inclusion in the sample as participating Schools met the
following criteria:• More than 10 teachers total• 80% or more of teachers viewed materials• Minimum average of 90.0 minutes of viewing per teacher for the school
– Districts included were only those for which eligible schools were included• Normalized for socio-economic and demographic factors
Data– Unit of measure was percent Proficient or Advanced as metric– Student performance data were captured from publically available, Internet-accessed
sources
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
Immediate Impacts
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
MATH:11.1 net PD 360 gain in Pct Adv+Prof (p<0.001)
Vs. 2.6 net gain for Districts
That equals:18.9% improved year-over-yearVs. 4.2% improvement for DistrictsThat’s 4.5 times MORE Improvement than their Districts
Reading:10.3 net PD 360 gain in Pct Adv+Prof (p<0.001)
Vs. 1.5 net gain for Districts
That equals:18.1% improved year-over-yearVs. 2.5% improvementThat’s 7.4 times MORE improvement than their Districts
7
4.9% gain for lower engagement schools (p<.01)
18.0% gain for higher engagement schools (p<.001)
Closed the Gap:267% advantage in gains for higher engagement schools (p<.001)
Nearly 4 times the impact
Predictors of Success
0.5% gain for lower engagement schools (p=ns)
18.9% gain for higher engagement schools (p<.001)
Surpassed the Gap:3,520% advantage in gains for higher engagement schools (p<.001)
36 times greater impact
Shaha SH, Ellsworth H (2014). Predictors of Success for Professional Development: Linking Student Achievement to School and Educator Successes through On-Demand, Online Professional Learning. Journal of Instructional Psychology (JIP), Vol. 40, No. 1
Predictors of Success
8
Higher Engagement
Schools
Lower Engagement
Schools Difference p-va lue
Percent Difference
Educator UtilizationSegments Viewed (school cumulative) 2298.0 528.9 1769.1 0.000 334.5% 4.3 Times
Programs Viewed 588.7 223.5 365.3 0.000 163.5% 2.6 Times
Minutes Viewed 359.9 180.6 179.3 0.000 99.3% 2.0 Times
Registered Users 87.7% 49.0% 38.7% 0.000 78.9% 1.8 Times
Registered Users Viewing at 90 min 79.1% 48.7% 30.4% 0.000 62.4% 1.6 Times
Forums Viewed 138.7 87.0 51.7 0.000 59.4% 1.6 Times
Total Users Licensed 93.9% 61.4% 32.5% 0.000 52.9% 1.5 Times
Educator EngagementFollow-up Questions Answered 359.6 167.0 192.6 0.000 115.3% 2.2 Times
Reflection Questions Answered 588.3 340.4 247.9 0.000 72.8% 1.7 Times
Files Uploaded 46.3 29.3 17.0 0.000 58.0% 1.6 Times
Focus Objectives Set Up 3120.7 2149.0 971.7 0.000 45.2% 1.5 Times
Files Downloaded 45.2 35.4 9.8 0.011 27.7% 1.3 Times
Forums Posted 28.5 23.4 5.1 0.031 21.8% 1.2 Times
Pct of Users in Communities 43.0% 36.5% 6.5% 0.045 17.8% 1.2 Times
Leadership, Implementation, AccountabilityObservations Performed 65.2 29.2 36.0 0.000 123.2% 2.2 Times
Focus Objectives Set Up (school cumulative) 3120.7 2149.0 971.7 0.000 45.2% 1.5 Times
Magnitude
Shaha SH, Ellsworth H (2014). Predictors of Success for Professional Development: Linking Student Achievement to School and Educator Successes through On-Demand, Online Professional Learning. Journal of Instructional Psychology (JIP), Vol. 40, No. 1
Intent: Quantify sustained impacts of PD360Participation
– Second Year follow-on study– Systematic sample of 169 elementary schools, in 73 districts, in 19 States– Same eligibility criteria for School versus District inclusion – Districts included were only those for which eligible schools were included
• Normalized for socio-economic and demographic factors
Data– Unit of measure was percent Proficient or Advanced as metric– Student performance data were captured from publically available, Internet-accessed
sources
Glassett, K. & Shaha, S. (2014). Lessons learned from a two year implementation project: Sustaining Student Gains with Online On-demand Professional Development. In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2014 (pp. 660-667). Chesapeake, VA: AACE.Glassett K, Shaha S & Ellsworth H. (2014). Are Student Gains Sustainable when using Online On-demand Professional Development? Association of Teacher Educators 2014 Summer Conference. Niagara Falls, NY: The Association of Teacher Educators. August 2014.
2-Year Sustained Impacts
2-Year Sustained ImpactsMATH vs. baseline:16.4 net PD 360 gain in Pct Adv+Prof (p<0.001)
13.5 more net gain than respective Districts
That equals:28.1% improved from baseline (p<0.001)
That’s 4.6 times MORE improvement than their Districts
Reading vs. baseline:17.2 net PD 360 gain in Pct Adv+Prof (p<0.001)
13.5 more net respective Districts (again)
That equals:30.2% improved from baseline (p<0.001)
That’s 5.0 times MORE improvement than their Districts
Glassett, K. & Shaha, S. (2014). Lessons learned from a two year implementation project: Sustaining Student Gains with Online On-demand Professional Development. In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2014 (pp. 660-667). Chesapeake, VA: AACE.Glassett K, Shaha S & Ellsworth H. (2014). Are Student Gains Sustainable when using Online On-demand Professional Development? Association of Teacher Educators 2014 Summer Conference. Niagara Falls, NY: The Association of Teacher Educators. August 2014.
7-Year Longevity
Intent: What is the long-term impact of PD 360 participation
Design– Quasi-experimental, longitudinal study– 25 states and 78 school districts– Each district is included only in the longevity duration class to which it
belongs
Shaha SH, Glassett K, Ellsworth H (2015). Long-term Impact of On-demand Professional Development on Student Performance: A Longitudinal Multi-State Study. Journal of International Education Research. Vol 11(1).Glassett, K. & Shaha, S. (2014). Long-term Impact of On-demand PD on Student Performance: A Longitudinal Multi-State Study. In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2014 (pp. 668-671). Chesapeake, VA: AACE.Shaha, S., Glassett, K., & Ellsworth, H. (2014). Long-term Impact of On-demand PD on Student Performance: A Longitudinal Multi-State Study. 2014 International Education Conference. San Francisco, CA: The Clute Institute. August 2014.
Change in Reading Perf Change in Math Perf0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
Longitudinal Effect on Change in Student Performance Versus Pre-PD Baselines
2-3 years4-5 years6-7 years
Net C
hang
e in
Pct
Adv
+Pro
f
7-Year Longevity
Reading gains from Baseline:2-3 years: 5.58 net gain in pct Adv+Prof
4-5 years: 7.34 net gain in pct Adv+Prof (p=0.042)
6-7 years: 12.77 net gain in pct Adv+Prof (p=0.037)
MATH gains from Baseline:2-3 years: 2.90 net gain in pct Adv+Prof
4-5 years: 6.92 net gain in pct Adv+Prof (p=0.031)
6-7 years: 12.15 net gain in pct Adv+Prof (p=0.009)
Shaha SH, Glassett K, Ellsworth H (2015). Long-term Impact of On-demand Professional Development on Student Performance: A Longitudinal Multi-State Study. Journal of International Education Research. Vol 11(1).
13
Executive SummaryPD360 Schools versus their Districts, Year-over-Year
– Math:• 11.1 Net gain in Math vs. 2.6 for District peers• That’s 18.9% better than Pre vs. 4.2% better for District peers• That’s 4.5 times more improvement than their peer Districts
– Reading:• 10.3 Net gain in Math vs. 1.5 for District peers• That’s 18.1% better than Pre vs. 2.5% better for District peers• That’s 7.4 times more improvement than their peer Districts
Predictors of Success– Math :
• 13.16 Net gain in for High Observation schools versus 2.22 net gain for Lower Observation Schools• That’s 24.9% improvement with High Observation & PD360 baseline versus 3.9% improvement for Lower Obs w/PD360• That’s 6.4 times more improvement for Higher Observation versus Lower
– Reading:• 12.47 Net gain in for High Observation schools versus 2.21 net gain for Lower Observation Schools• That’s 24.6% improvement with High Observation & PD360 baseline versus 3.5% improvement for Lower Obs w/PD360• That’s 7.0 times more improvement for Higher Observation versus Lower
2-Year Sustained Advantage versus their Districts– Math :
• 16.4 Net gain in Math from baseline – that’s 13.5 more net gain than District peers• That’s 28.1% better than baseline, which is 4.6 times more improvement than their peer Districts
– Reading:• 17.2 Net gain in Math from baseline – that’s 13.5 more net gain than District peers (matches Math)• That’s 30.2% better than baseline, which is 5.0 times more improvement than their peer Districts
7-Year Longevity: into 6th and 7th Years– Math net gains:
• 5.58 for educational organizations with 2-3 years of PD• 7.34 for orgs with 4-5 PD years • 12.77 for orgs with 6-7 PD years
– Reading net gains:• 2.90 for educational organizations with 2-3 years of PD• 6.92 for orgs with 4-5 PD years • 12.15 for orgs with 6-7 PD years
14
Executive Summary
Next Studies• Aligned Observation• Aligned Seminars• Aligned Implementation
4. Advantages with Longevity into 6th and 7th Years3. Advantages Sustained into Second Year of PD 3602. Prdictors of Success
High vs. Lower Engagement1. Advantage with PD 360
versus none, Year-over-Year
15
Soon to be Published
Observation Integrated– Math :
• 13.16 Net gain in for High Observation schools – versus 2.22 net gain for Lower Observation Schools
• 24.9% improvement with High Observation & PD360 baseline – versus 3.9% improvement for Lower Obs w/PD360
• That’s 6.4 times more improvement for Higher Observation versus Lower– Reading:
• 12.47 Net gain in for High Observation schools – versus 2.21 net gain for Lower Observation Schools
• 24.6% improvement with High Observation & PD360 baseline – versus 3.5% improvement for Lower Obs w/PD360
• That’s 7.0 times more improvement for Higher Observation versus Lower
16
Key Impacts:Immediate Impacts
2-Year Sustained Impacts7-Year Longevity
Predictors of Success
Prepared by: Steven H. Shaha, PhD, DBA