Upload
sarahpespark
View
214
Download
0
Embed Size (px)
Citation preview
7/22/2019 Student Engagement in Educational Apps
1/7
STUDENT ENGAGEMENTIN EDUCATIONAL APPS
A white paper by Maya Lopuch, Data Scientist at eSpark Learning
Copyright 2013, eSpark Learning
7/22/2019 Student Engagement in Educational Apps
2/7
As iPads enter the classroom, they bring withthem an unprecedented number of educational
resources. Developers have created thousands
of highly engaging apps that target specic
academic skills. With so many educational apps
on the App Store, the Apple website explains,
theres no limit to your learning.
In practice, one limit may be the capability of
educators to select apps for their classrooms.
A search for fractions yields 506 different
iPad apps alone. Over 100 of those apps have
a four star rating or higher. While this level of
diversity can be a boon for students, the task foreducators is daunting. It is easy to be paralyzed
by choice.
Educators have some resources to guide them
in these decisions. On the App Store, educators
can view screenshots of apps and read through
reviews before purchasing an app. However,
teachers and parents almost always write these
reviews. The views of students, the target
audience, are not directly represented. If an app
does have high reviews, its not immediately clea
whether that is because it works well for parentsor if its a good t for the classroom. Apps with
in-app purchases, or apps that require a lot
of manual setup and adult directions, may be
good for one-on-one but may not work well in a
classroom environment.
Educators intuition remains the most valuable
resource in evaluating the quality of content.
Many agree that interactive apps create
engaging learning experiences. Apps that have a
worksheet-type interface and focus on repetitiontend to be less engaging for students. Applying
those kinds of heuristics can help reduce a list
of over 100 four-star fraction apps to a more
manageable list of candidates.
New data is enabling educators to take the
app selection process several steps further.
This paper describes how eSpark Learning is
using a unique database of student sentiment
to identify highly engaging iPad apps. This data
builds upon the subjective expertise of teachersin two important ways. First, eSpark draws from
millions of ratings from thousands of students
to determine which apps students nd more
engaging than others. Second, eSpark can
identify the specic app characteristics that are
predictive of student engagement.
These analytical insights are changing how
educators incorporate technology into the
classroom. Personalization engines like those
Figure 2:These two apps, iTooch 5th Grade Math and 5
Dice: Order of Operations Game, teach similar skills but differ
dramatically in their approach. iTooch encourages students
to practice rote skills and 5 Dice focuses on higher-order
thinking.
Figure 1: A search for fractions in the App Store yields 506
different iPad apps.2
7/22/2019 Student Engagement in Educational Apps
3/7
offered by Netix and Amazon have changed
how consumers nd products in entertainment
and retail. To date, education has largely
eschewed personalization. The most prevalent
model in K-12 continues to be one instructor
teaching many students the same lesson in
the same way. Mobile devices now allow for
individualized delivery of content, and data on
student engagement can help personalize that
content to student preferences. Educators are
now better equipped to augment traditional
classroom instruction with a personalized
learning experience.
Building a rich database of educational apps
eSpark Learning is a personalized learning
platform for the iPad. Pedagogical experts
create personalized learning curricula using
curated instructional videos, third-party apps,
and assessment tools. Each curriculum is closelyaligned to a Common Core domain and grade
level. Thousands of students across the country
use eSpark to augment typical classroom
instruction in math and reading in grades K-8.
Students work through a curriculum that is
tailored to their academic needs and learning
goals. For example, one student may work on
second grade level Number and Operations in
Base Ten while her classmate may work on fth
grade level Geometry.
To curate this digital content, eSparks curriculum
design experts scour the App Store to nd the
best educational resources that are aligned
with the Common Core State Standards. Each
potential app is evaluated on a rubric that
considers Common Core standard alignment,
authenticity of task, scaffolding of learning,intuitiveness, student engagement, and cost
of the app. Educational apps that receive high
scores on this rubric are incorporated into the
eSpark curricula. This process is a rigorous
and large-scale solution to the challenges that
teachers face when searching for apps for their
own classrooms.
Once apps are selected to be in the eSpark
curriculum, curriculum designers catalogue
each app on whether it ts dozens of additionalcategories. These categories cover a wide
variety of app characteristics pertaining to
content, pedagogical framework, and interface.
For example, each app is examined for social
studies applications, text highlighting capabilities,
and gyromotion. This extensive attribute tagging
process provides a highly granular view into the
composition of educational apps.
This attribute database is continually expanding
as new content enters the eSpark curriculum.
Figure 3:The eSpark rubric has six categories of app quality.
3
7/22/2019 Student Engagement in Educational Apps
4/7
This paper highlights a subset of the 2012-2013
eSpark curriculum. In total, 395 apps covering
content from grade levels PK-5 are reviewed in
this study. Table 1 shows counts of the most
common app attributes in this sample.
Linking apps to student engagement
eSpark Learning links app characteristics tostudent engagement with ordinary least squares
multiple regression models. App attributes are
the independent variables in the prediction
equation. The dependent variable, student
engagement, comes from a database of
student sentiment. As students work through
their personalized learning plan in eSpark, they
are asked to rate each of the curated apps
they explore. Students rate each app with a
thumbs up or thumbs down. Although this
binary indicator of sentiment is an imperfect
proxy of student engagement, the large sample
sizes of ratings help to capture more accurate
information. The 395 apps analyzed in this study
have been rated by 1,203 students.1
1The apps and students represented in this sample are a subset of all of the apps in the eSpark curriculum and total students using
eSpark. This subset was selected to be analyzed rst because of the grade level of content. eSpark will extend the sample as the attribute
catalogue grows to encompass apps from grade levels 6-8.
Attribute Defnition Apps with Attribute in Sample
Sound Effects The app has interesting, noticeable, and motivating sound effects. 241
Graphics, colors, andsupplemental visuals Graphics and supplemental visuals are colorful and visuallystimulating. 201
Multiple-activity appThe app has multiple activities (i.e. adding fractions, comparingfractions) and requires directions from eSpark. 183
Moderate scaffoldingUsers are allowed to try again but receive no feedback explainingwhy their answer was wrong. 182
InteractiveThe user interacts with the app by tapping or swiping in responseto an academic question. 179
One-activity appThe app has one activity. Students are able to go right into the appwithout having to locate the activity. 140
Female voice Some or all of the app is narrated by a woman. 122
Cartoon/animal characterCartoon characters (people, animals, monsters) are part of the app;app has an animal graphic. 116
Background music The app has music in the background. 109
Score/progress overviewAt the end of the module, chapter, or game students see anoverview of their progress, e.g. how many questions theyanswered correctly.
94
Table 1:Ten of the most popular attributes in eSpark PK-5 curated apps. The prediction models described in this paper
included a more extensive set of regressors.
4
7/22/2019 Student Engagement in Educational Apps
5/7
Among apps analyzed in this
sample, the mean student
engagement rating is 82%.This high average rating
reects the fact that the apps
in this sample have been highly
curated with expected student
engagement as one criterion. A
random sample of educationalapps would likely have a lower
average engagement rating.
eSpark then calculates the
share of positive ratings to
total ratings to determine an
engagement metric for each
app in the curriculum. This
average rating is the dependent
variable in the regression
models, and app features are
the primary regressors. These
models also control for the
grade level and subject area of
the content.
Findings
The results from the prediction
equations are shown in Table
2. Among the curated set of
395 apps covering content for
grade levels PK-5, we show
the app characteristics thatare most strongly predictive
of student engagement. The
effect on student engagement
Attribute Defnition Effect on Student Engagement p-value
AnimationApp includes animation of graphicallycreated characters. +2.2% 0.029
Background music App has music in the background. +1.9% 0.021
Badges or rewardsStudents are rewarded for correct answerswith badges or rewards. +1.7% 0.087
Bare-bones interfaceInterface is black/white/green, low-budget, and does not boast stimulatinggraphics.
2.3% 0.039
Complex menu ofactivities
Menu of activities has many differentfunctions and may be difcult to navigate. 2.5% 0.055
Drill and kill Students practive the skill in a rote andrepetitive way. 2.5% 0.032
Graphics, colors, andsupplemental visuals
Graphics and supplemental visuals arecolorful and visually stimulating. +2.1% 0.003
SilentNo sound effects, music, or audio areincluded. 4.6% 0.001
Figure 4: Students are asked to rate each app in their personalized curriculum with
a thumbs up or a thumbs down button. The average app in this sample received
82 thumbs up for every 100 student ratings.
Table 2: The app characteristics that are most strongly predictive of student engagement among the curated set of 396 appscovering content for grade levels PK-5. 5
7/22/2019 Student Engagement in Educational Apps
6/7
can be interpreted as the
contribution of a particular
attribute, holding all else equal,
to the average rating of apps
in this sample. For example,
among two otherwise similar
apps, an app with background
music is predicted to have a
mean student rating that is two
percentage points higher than
an app without music.
These results provide empirical
support for three main ndings.
First, students are very
sensitive to audio and visual
components of apps. Most
of the strongest predictors of
student engagement relate to
the app interface. Compared
to otherwise similar apps,
apps that have distinguishing
animation, music, or graphics
are predicted to have higher
student engagement ratings.
Apps that are silent or have
a bare-bones interface are
predicted to have signicantly
lower ratings than average
apps. This data suggests thatstudent engagement in iPad
apps is in large part driven by
whether apps can differentiate
the experience from a more
typical educational exercise.
Students gravitate toward rich
media and this is consistently
reected in their engagement
ratings.
Second, the data on studentengagement conrms some
common hypotheses on
pedagogical strategies.
Students respond positively
to badges and rewards. Apps
that reward students for correct
answers are associated with
an engagement rating that is
two percentage points higher.
Apps that follow a Drill and kill
strategy are associated with
ratings that are 2.5 percentage
points lower than average
apps. Although the teaching
medium changes when content
is delivered through an iPad
app, student responses to
pedagogical strategies remain
largely consistent. Student
motivation remains a central
component of the educational
experience.
Third, apps that offer a large
degree of choice reduce
engagement. Apps that offer a
complex menu of activities are
associated with signicantly
lower student engagement
ratings. This nding may
come as a surprise to many
educators. Several of the
most popular apps in the
App Store advertise extensive
functionalities. Educators often
gravitate to these omnibus
Figure 5: The app Fractions. Smart Pirate engages students with animated cartoons,
music, and stimulating graphics.
Figure 6: The app Counting Caterpillar, which teaches fundamental counting skills,
rewards performance with badges and rewards.6
7/22/2019 Student Engagement in Educational Apps
7/7
apps because they offer so much content in
one package. That these comprehensive apps
are associated with reduced engagementsuggests that students may feel overwhelmed
by the upfront decisions required of these apps.
Clarity of use has a strong impact on student
engagement.
Whats Next
By linking qualitative characteristics of apps
to student engagement, eSpark has identied
empirical insights that were otherwise
unattainable at scale. This new knowledge hasled eSpark to rene its app curation process.
A deeper understanding of what kinds of apps
resonate with students has enabled eSpark
curriculum experts to make more sophisticated
design decisions. The newest eSpark curriculum
reects many of the insights described here.
eSpark continues to prioritize educational rigor
as the foundation of its curriculum, but now
curriculum experts are able to select content
with a much stronger understanding of how to
identify the most engaging apps.
eSpark hopes this data will reach further and
inform decisions among both educators and
app developers. When educators evaluate apps,
they would be wise to prioritize content that has
a rich interface, adheres to proven pedagogical
strategies, and prioritizes intuitiveness of use.
App developers can apply these lessons to
create more content that meets these best
practices.
The attributes discussed here merely scratch the
surface of the universe of app characteristics.
eSpark is continuing to develop its appdatabase by expanding attribute tagging in both
breadth and depth. Newly developed apps are
continually added to this catalogue, and apps in
the current eSpark curriculum are being tagged
for even more extensive attributes.
Looking forward, eSpark will use this attribute
database to develop a richer understanding
of how iPad apps impact student outcomes.
This study has explored the effects of app
characteristics on student engagement. Inthe future, eSpark will be able to link specic
app characteristics to impacts on student
achievement. This data will allow educators to
rene app curation even further, enabling them
to select the content that has been proven to
be most effective for improving student learning.
Looking forward even further, eSpark is excited
about a future where educators will be able to
match apps to students, at scale, based on
students interests, preferences, and motivations.
Figure 7: The apps Coin Math and iTooch offer many applications but require students to choose from a complex menu before
engaging with activities.
Maya LopuchData Scientist at eSpark Learning
Maya Lopuch leads research and analytics
at eSpark Learning. Prior to joining eSpark,
she was a researcher at the Harvard
Graduate School of Education, where
her work investigated how public schools
impact inequality and long term student
outcomes. Maya holds degrees in Economics and Public Policy
from Stanford University and the University of Chicago. She
believes in using data to inform and improve education. 7