Upload
others
View
19
Download
0
Embed Size (px)
Citation preview
Best Practices for Evaluating
Mobile Learning:
Real Stories from Around the Globe
Dr. Julie A. EvansCEO, Project Tomorrow June 25, 2018 @SpeakUpEd@JulieEvans_PT
Dr. Julie A. EvansChief Executive Officer, Project Tomorrow
Chief Researcher, Speak Up Research Project
Mobile Learning Project Evaluator since 2007
Also, skilled in the evaluation of blended learning and digital content usage in K-12 and higher education
Thank you for being here!
Special Guest:
Hiram Lopez-LandinSenior Analyst, Wireless ReachQualcomm
Today’s discussion topics
▪ About our work o Project Tomorrow and the Speak Up Research Project o Qualcomm Wireless Reach
▪ Why are evaluations of mobile learning important today?
▪ State of mobile learning today & evaluations for mobile learning
▪ Let’s talk about 3 different mobile learning projects
▪ My top 8 tips and strategies for evaluating mobile learning
▪ Extended learning – links to some additional resources
▪ Your thoughts, insights, questions
Tweet with us!
@JulieEvans_PT @SpeakUpEd@hlopezlandin @QCWirelessReach
Getting to know you!
About Project Tomorrow (www.tomorrow.org)
▪ Nonprofit education organization supporting K-12 education since 1996
▪ Mission is to ensure all of today’s students are well prepared for the future
▪ Programs and research focus on role of digital tools within the education ecosystem – believe in power of STEAM to support student preparation for college and career success
o Speak Up Research Project on Digital Learning: collecting & reporting on the authentic feedback of K-12 stakeholders to inform federal, state & local programs and policies
About the Speak Up Project (www.tomorrow.org/speakup)
▪ Annual research project since 2003
▪ Uses online surveys + focus groups
▪ Facilitated 100% through schools and districts
▪ We design online surveys to collect feedback from your K-12 Students, Teachers, Parents, Administrators, and Community Members
▪ All K-12 schools – public, private, parochial, charter, virtual - are eligible to participate
▪ Project Tomorrow manages all data collection and reporting for you - 100% free service
▪ Schools get summary report with all locally collected data + state and national data for benchmarks
We share national data with federal, state and local policymakers to inform programs and funding
About the Speak Up Project (www.tomorrow.org/speakup)
Survey question topics include:
✓ Use of technology to support learning ✓ School climate for innovation ✓ College and career ready skill development/interest ✓ Leadership challenges ✓ Teachers’ needs for professional learning✓ How do different stakeholders value digital learning ✓ Emerging trends w/digital tools, content and resources ✓ New classroom models: mobile, blended, flipped✓ School to home communications✓ Designing the ultimate school
❖ Activities
❖ Attitudes
❖ Aspirations
Since 2003, 5.4 million K-12 stakeholders have submitted a Speak Up survey
10
Qualcomm Wireless Reach
11
45
PROGRAMS
17
COUNTRIES
150
STAKEHOLDERS
270K
PEOPLE
Wireless Reach Education Portfolio Impact to Date
12
17
9
7
6
2
2
1
1
United States
East Asia
Southeast Asia
South Asia
Middle…
Latin America
Sub-Saharan…
Europe
Programs by RegionPercent of female beneficiaries in all education programs
49%
Organizations served in education portfolio570
Wireless Reach Education 10 Years of Accomplishments
Devices used in education portfolio11,000
All figures include aggregated data from 2006 to 2017
13
Education Strategy
Wireless Reach education
programs are designed to address
the barriers to adoption of wireless
technology in the classroom
including the need for:
• Digital content & assessment
• Infrastructure & IT support
• Privacy & security
Why is it important to evaluate our mobile learning projects?
Why is it important to evaluate our mobile learning projects?
Are our investments yielding a return on investment?
What do you see?
Are we all seeing
“evaluation” or even
“mobile learning”
the same way?
State of Mobile Learning: selected research findings from Speak Up 2017
State of Mobile Learning: selected research findings from Speak Up 2017
Learning on the go – anytime, anywhere
Survey Audience # of Surveys Submitted
K-12 Students 340,927
Teachers & Librarians 34,833
Parents 23,159
Administrators 3,249
Community Members 4,611
About schools and districts: 3,641 districts, 10,619 schools29% urban, 34% rural, 37% suburban, 68% title 1 eligible
National participation in Speak Up 2017: 406,779
@Project Tomorrow 2017
Students have more personal access to mobile devices than ever before
35%
50%
79%
91%
53%57% 55%
39%32% 35%
48%54%
10%15% 14% 13%15% 12%
K-2 Gr 3-5 Gr 6-8 Gr 9-12
Smartphone Tablet Laptop Digital Reader Smartwatch
@Project Tomorrow 2017
Teachers and mobile devices within learning
Classroom access is changing rapidly
In class access
2015 2017
BYOD 17% 23%
1:1 in school assignment
20% 31%
1:1 in school + home assignment
12% 14%
Cart access 31% 30%
No access 31% 18%
Not surprising: teachers want
more PD on how to
effectively integrate mobile
devices within instruction
@Project Tomorrow 2017
Administrators say mobile learning = positive results
▪ 49% of school principals say they see positive academic results from their 1:1 mobile learning programs
Benefits of mobile learning Elementary school
principals
Middle school principals
High school principals
Students develop topics more fully 82% 83% 77%
Students are more engaged in learning 72% 62% 55%
Preparation for college/work 71% 71% 71%
Increased access to online content 69% 73% 73%
Learning is more personalized 65% 60% 57%
Students are collaborating more 57% 58% 58%
Improved teacher-student communications 48% 58% 63%
But what metrics are school and district leaders using to measure outcomes from mobile learning?
Turn to a neighbor and share: How are you evaluating the success of your mobile learning programs?
How satisfied are you with your evaluation process?
Audience feedback
Administrators say these are the top ways they are measuring the impact of their digital initiatives:
Metrics % of district administrators
nationwide
Which ones are you using?
Student engagement levels 77%
Classroom observations 69%
Student feedback 62%
Student achievement results 60%
Teacher buy-in 54%
Teacher feedback 53%
Parent feedback 37%
Two of the 8 essentials:
✓Purposeful planning for mobile device usage
✓Measuring project results with meaningful metrics
✓Purposeful planning for mobile device usage
▪ What are your goals?▪ What outcomes do
you want to see? ▪ Why is this important
for your school or district?
✓Measuring project results with meaningful metrics
▪ Aligning metrics with goals
▪ Leveraging appropriate tools
▪ Being open to what you don’t know yet
Evaluating mobile learning begins with project planning
Introduction to 3 mobile learning projects, the metrics they used/are using to evaluate program success & lessons learned that can inform your efforts:
❑ “Making Learning Mobile” – Chicago
❑ “STEAMing Ahead with Mobile Learning – San Diego
❑ “Classroom of the Future” – Turkey
Let’s get real!
“Making Learning Mobile” – Chicago
o Three year focused study on the impact of extended Internet access and mobile learning on student achievement
o Study group: 5th grade students and their teachers at Falconer Elementary School in Chicago, IL
o Students each assigned a Samsung Android tablet with 4G LTE enabled education broadband to use at school and at home
Partners: Kajeet, Qualcomm Wireless Reach, Project Tomorrow
“Making Learning Mobile” – Chicago
Objectives of the project was to learn more about:
✓ How did the students and teachers use the tablets and extended Internet connectivity for learning?
✓ What benefits did they derive from those experiences?✓ What was the impact of the experiences on student learning and overall
academic achievement?
Evaluation metrics included:
• Feedback data from students and teachers – analysis of changes over 3 years • Classroom observations to evaluate depth of usage and impact of home access • Analysis of student achievement results
“Making Learning Mobile” – Chicago
Key findings from the study:
1. Tablet became an indispensable learning tool
2. Tablets and extended connectivity enabled greater communications between students and teachers
3. Benefits to students included development of college/career ready skills and new learning behaviors
4. Teachers’ skills, motivation and readiness for tablet integration impacted effectiveness of usage
5. Purposeful integration within math = higher student achievement and growth
“Making Learning Mobile” – Chicago
Insights from the evaluation of this project:
▪ Build your evaluation methodology and tools to be flexible and open to “surprises”
▪ Understand that the teacher adoption/assimilation/adaption process may be a three year adventure – be reasonable with project goals and the timeline for intended outcomes, and adjust your logical framework accordingly
▪ Reality alert: traditional data collection and analysis tools may not work for mobile learning
▪ With mobile learning, it is essential to collect and analyze both quantitative and qualitative data – feedback matters
▪ Keep eyes wide open for lessons learned – these are gold!
“STEAMing Ahead with Mobile Learning” – San Diego
Partners: San Diego Public Library, e3 Civic High School, Qualcomm Wireless Reach, Project Tomorrow
9th graders were provided with 4GL tablet that included an AR app to learn about the STEAM elements within the Central Library Dome design, engineering and construction
“STEAMing Ahead with Mobile Learning” – San Diego
Objectives of the project was to learn more about:
✓ How a contextually relevant learning experience for students could introduce students to the role of STEAM within construction and design
✓ The role of tablets and 4GL in increasing student engagement in learning✓ How to effectively increase student interest in STEAM career fields ✓ The ways in which this application could be used within other curricular fields
and with the general public
Evaluation metrics included:
• Feedback data from students and teachers• Observation of the learning activity with “person on the street” interviews • Analysis of changes in students’ interest levels in STEAM career fields
“STEAMing Ahead with Mobile Learning” – San Diego
Key findings from the study:
1. Students said use of AR content increased their engagement in learning
2. Benefits included having opportunity to work collaboratively with classmates on a real world activity; enjoyment level surprised the students!
3. Different learning modalities within the experience appealed to different students: videos, images, text, assessment
4. 40% of the were more interested in a STEAM career field as a result of this learning experience
5. Teachers’ excited about how to use AR in other curricular areas; library staff exploring ways to bring same content to general public
“STEAMing Ahead with Mobile Learning” – San Diego
Insights from the evaluation of this project:
▪ A small project evaluation period demands tighter adherence to project goals. Project goals therefore must be clearly defined with broad buy-in from all stakeholders and partners.
▪ Critical to use the right tools to collect data. Example: to show change in attitudes, leverage pre and post data collection to “see” process of discovery and new ideas
▪ Reality alert: students are not familiar with all new technologies and thus, the awareness/tool learning process must be taken into account within the analysis of impact
▪ With a new technology such as AR, bring teachers into the evaluation development process upfront so that the analysis of the outcomes is based upon a realistic understanding of the environment
“Classroom of the Future” – Turkey
Partners: Turkish Education Association, Qualcomm Wireless Reach
▪ 7th grade students and their teachers from 7 schools have Qualcomm-enabled tablets with wireless connectivity to support learning in school and at home
▪ Initial focus was on English instruction classes but project has expanded to include other subject areas now
▪ Teachers were provided with specialized professional development to support this project
Objectives of the project was to learn more about:
✓ How the inclusion of mobile devices in the learning environment enhances student learning and engagement particularly in reading and writing
✓ How the mobile learning experience improves students’ perceptions of their technology skills especially as an important global workplace skill
✓ If the teacher PD and the use of the devices in the classroom results in increases in teachers’ capacity, comfort and confidence in implementing mobile learning
Evaluation metrics included:
• Feedback data from students and teachers – pre and post surveys • Aligning pre and post survey data elements directly with project goals • Linking teacher data with student data for analysis of impact
Key findings from the study (so far!):
1. Students and teachers agree: the use of the tablets with wireless connectivity both at school and at home are important to support learning.
2. 3/4 of the teachers reported: mobile learning environment resulted in improved student engagement, enhanced quality of their students’ work and students taking greater responsibility for their own learning.
3. The availability of the 1:1 tablets for student use at school and at home increased the students’ overall use of technology for learning - 86% of the students developed stronger technology skills during the school year.
4. With the new tools and training, 88% of the teachers reported greater communications with their students and 76% noted an increase in their overall effectiveness as a teacher.
Interim insights from the evaluation of this project:
▪ Even as project expands to include new teachers and schools, it is important to keep the objectives and evaluation plan stable so that you can evaluate for growth and maturity alongside the success of the replication.
▪ Analyzing teacher and student data at the class/school level side by side provided valuable insights into the relationship between usage, personal valuation and impact.
▪ Monitoring is not the same as evaluation. But monitoring is important for a valid evaluation.
▪ A good evaluation demands good data. To get good data, you have to ask the right questions.
Selection of tips and strategies you should think
about when evaluating your own mobile learning
projects
8
Selection of tips and strategies to use when evaluating your own mobile learning projects
1. Evaluation is not magic. Write out a plan and do that at the same time you are developing your operational plan for implementation so it aligns to your objectives for your project .
Selection of tips and strategies to use when evaluating your own mobile learning projects
1. Evaluation is not magic. Write out a plan and do that at the same time you are developing your operational plan for implementation so it aligns to your objectives for your project .
2. The quality of the data will determine the validity of your evaluation – and thus, what you will learn. You should always worry about the validity of your data!
Selection of tips and strategies to use when evaluating your own mobile learning projects
1. Evaluation is not magic. Write out a plan and do that at the same time you are developing your operational plan for implementation so it aligns to your objectives for your project .
2. The quality of the data will determine the validity of your evaluation – and thus, what you will learn. You should always worry about the validity of your data!
3. Use the right tool for the right objectives in your plan. Some tools work better than others for certain objectives.
Selection of tips and strategies to use when evaluating your own mobile learning projects
1. Evaluation is not magic. Write out a plan and do that at the same time you are developing your operational plan for implementation so it aligns to your objectives for your project .
2. The quality of the data will determine the validity of your evaluation – and thus, what you will learn. You should always worry about the validity of your data!
3. Use the right tool for the right objectives in your plan. Some tools work better than others for certain objectives.
4. Evaluation is ongoing – full year and year to year
Selection of tips and strategies to use when evaluating your own mobile learning projects
5. Just because you use a mobile device and you are in education don’t assume that you know how to evaluate a mobile learning project. Evaluation is tricky business to get right!
Selection of tips and strategies to use when evaluating your own mobile learning projects
5. Just because you use a mobile device and you are in education does not mean that you automatically know how to evaluate a mobile learning project. This is tricky business to get right!
6. Keep your eyes wide open for surprises and unintended outcomes –mobile learning is not “jello” yet. This field is still developing and there is no recipe or formula for success yet.
Selection of tips and strategies to use when evaluating your own mobile learning projects
5. Just because you use a mobile device and you are in education does not mean that you automatically know how to evaluate a mobile learning project. This is tricky business to get right!
6. Keep your eyes wide open for surprises and unintended outcomes –mobile learning is not “jello” yet. This field is still developing and there is no recipe or formula for success yet.
7. Be realistic about your outcomes especially regarding the timeframes in which you expect to see those outcomes. To get to the big impact outcomes takes a lot of work in the classroom!
Selection of tips and strategies to use when evaluating your own mobile learning projects
8. Most importantly, an evaluation of a mobile learning project is about so much
more than mobile devices. It is really about the transformative learning
experiences at your school.
Take advantage of that!
More resources available for extended learning
https://www.tomorrow.org/publications/MakingLearningMobile.html▪ 3 years of evaluation reports from the Making Learning Mobile project
https://www.qualcomm.com/media/documents/files/united-states-making-learning-mobile-mlm-project.pdf▪ Case study on the Qualcomm Wireless Reach site
Additional information about the projects featured today:
Making Learning Mobile – Chicago
More resources available for extended learning
Additional information about the projects featured today:
STEAMing Ahead with Mobile Learning – San Diego
https://www.qualcomm.com/media/documents/files/united-states-steaming-ahead-with-mobile-learning.pdf
Classroom of the Future – Turkey
(not published yet – hard copies provided today for you!)
More resources available at www.tomorrow.org
National Speak Up reports and infographics
Targeted and thematic reportsDigital learning trendsCommunity engagement Mobile learning – infographic! Games in the classroomBlended learning outcomes
Presentations, podcasts and webinars
Services: consulting, workshops, evaluation and efficacy studies✓ We have expertise in the evaluation of mobile implementations, new
classroom models and digital content usage
New Speak Up 2017 reports released thru Sept 2018
Plan now for your district to participate in Speak Up 2018
Online surveys for: o K-12 students – individual + groupo Parents – English and Spanisho Teachers o Librarians/Media Specialistso School Site & District Administratorso Technology Leaderso Community Members &
Communications Officers
Surveys open Oct 2018 – Jan 2019Learn more www.tomorrow.org/speakup
100% free service to all schools and districts – including reports with comparison data
Your questions,
comments,
thoughts
Best Practices for Evaluating
Mobile Learning:
Real Stories from Around the Globe
Dr. Julie A. EvansCEO, Project Tomorrow June 25, 2018 @SpeakUpEd@JulieEvans_PT