Student Engagement in Educational Apps

7/22/2019 Student Engagement in Educational Apps

1/7

STUDENT ENGAGEMENTIN EDUCATIONAL APPS

A white paper by Maya Lopuch, Data Scientist at eSpark Learning

Copyright 2013, eSpark Learning


2/7

As iPads enter the classroom, they bring withthem an unprecedented number of educational

resources. Developers have created thousands

of highly engaging apps that target specic

academic skills. With so many educational apps

on the App Store, the Apple website explains,

theres no limit to your learning.

In practice, one limit may be the capability of

educators to select apps for their classrooms.

A search for fractions yields 506 different

iPad apps alone. Over 100 of those apps have

a four star rating or higher. While this level of

diversity can be a boon for students, the task foreducators is daunting. It is easy to be paralyzed

by choice.

Educators have some resources to guide them

in these decisions. On the App Store, educators

can view screenshots of apps and read through

reviews before purchasing an app. However,

teachers and parents almost always write these

reviews. The views of students, the target

audience, are not directly represented. If an app

does have high reviews, its not immediately clea

whether that is because it works well for parentsor if its a good t for the classroom. Apps with

in-app purchases, or apps that require a lot

of manual setup and adult directions, may be

good for one-on-one but may not work well in a

classroom environment.

Educators intuition remains the most valuable

resource in evaluating the quality of content.

Many agree that interactive apps create

engaging learning experiences. Apps that have a

worksheet-type interface and focus on repetitiontend to be less engaging for students. Applying

those kinds of heuristics can help reduce a list

of over 100 four-star fraction apps to a more

manageable list of candidates.

New data is enabling educators to take the

app selection process several steps further.

This paper describes how eSpark Learning is

using a unique database of student sentiment

to identify highly engaging iPad apps. This data

builds upon the subjective expertise of teachersin two important ways. First, eSpark draws from

millions of ratings from thousands of students

to determine which apps students nd more

engaging than others. Second, eSpark can

identify the specic app characteristics that are

predictive of student engagement.

These analytical insights are changing how

educators incorporate technology into the

classroom. Personalization engines like those

Figure 2:These two apps, iTooch 5th Grade Math and 5

Dice: Order of Operations Game, teach similar skills but differ

dramatically in their approach. iTooch encourages students

to practice rote skills and 5 Dice focuses on higher-order

thinking.

Figure 1: A search for fractions in the App Store yields 506

different iPad apps.2


3/7

offered by Netix and Amazon have changed

how consumers nd products in entertainment

and retail. To date, education has largely

eschewed personalization. The most prevalent

model in K-12 continues to be one instructor

teaching many students the same lesson in

the same way. Mobile devices now allow for

individualized delivery of content, and data on

student engagement can help personalize that

content to student preferences. Educators are

now better equipped to augment traditional

classroom instruction with a personalized

learning experience.

Building a rich database of educational apps

eSpark Learning is a personalized learning

platform for the iPad. Pedagogical experts

create personalized learning curricula using

curated instructional videos, third-party apps,

and assessment tools. Each curriculum is closelyaligned to a Common Core domain and grade

level. Thousands of students across the country

use eSpark to augment typical classroom

instruction in math and reading in grades K-8.

Students work through a curriculum that is

tailored to their academic needs and learning

goals. For example, one student may work on

second grade level Number and Operations in

Base Ten while her classmate may work on fth

grade level Geometry.

To curate this digital content, eSparks curriculum

design experts scour the App Store to nd the

best educational resources that are aligned

with the Common Core State Standards. Each

potential app is evaluated on a rubric that

considers Common Core standard alignment,

authenticity of task, scaffolding of learning,intuitiveness, student engagement, and cost

of the app. Educational apps that receive high

scores on this rubric are incorporated into the

eSpark curricula. This process is a rigorous

and large-scale solution to the challenges that

teachers face when searching for apps for their

own classrooms.

Once apps are selected to be in the eSpark

curriculum, curriculum designers catalogue

each app on whether it ts dozens of additionalcategories. These categories cover a wide

variety of app characteristics pertaining to

content, pedagogical framework, and interface.

For example, each app is examined for social

studies applications, text highlighting capabilities,

and gyromotion. This extensive attribute tagging

process provides a highly granular view into the

composition of educational apps.

This attribute database is continually expanding

as new content enters the eSpark curriculum.

Figure 3:The eSpark rubric has six categories of app quality.

3


4/7

This paper highlights a subset of the 2012-2013

eSpark curriculum. In total, 395 apps covering

content from grade levels PK-5 are reviewed in

this study. Table 1 shows counts of the most

common app attributes in this sample.

Linking apps to student engagement

eSpark Learning links app characteristics tostudent engagement with ordinary least squares

multiple regression models. App attributes are

the independent variables in the prediction

equation. The dependent variable, student

engagement, comes from a database of

student sentiment. As students work through

their personalized learning plan in eSpark, they

are asked to rate each of the curated apps

they explore. Students rate each app with a

thumbs up or thumbs down. Although this

binary indicator of sentiment is an imperfect

proxy of student engagement, the large sample

sizes of ratings help to capture more accurate

information. The 395 apps analyzed in this study

have been rated by 1,203 students.1

1The apps and students represented in this sample are a subset of all of the apps in the eSpark curriculum and total students using

eSpark. This subset was selected to be analyzed rst because of the grade level of content. eSpark will extend the sample as the attribute

catalogue grows to encompass apps from grade levels 6-8.

Attribute Defnition Apps with Attribute in Sample

Sound Effects The app has interesting, noticeable, and motivating sound effects. 241

Graphics, colors, andsupplemental visuals Graphics and supplemental visuals are colorful and visuallystimulating. 201

Multiple-activity appThe app has multiple activities (i.e. adding fractions, comparingfractions) and requires directions from eSpark. 183

Moderate scaffoldingUsers are allowed to try again but receive no feedback explainingwhy their answer was wrong. 182

InteractiveThe user interacts with the app by tapping or swiping in responseto an academic question. 179

One-activity appThe app has one activity. Students are able to go right into the appwithout having to locate the activity. 140

Female voice Some or all of the app is narrated by a woman. 122

Cartoon/animal characterCartoon characters (people, animals, monsters) are part of the app;app has an animal graphic. 116

Background music The app has music in the background. 109

Score/progress overviewAt the end of the module, chapter, or game students see anoverview of their progress, e.g. how many questions theyanswered correctly.

94

Table 1:Ten of the most popular attributes in eSpark PK-5 curated apps. The prediction models described in this paper

included a more extensive set of regressors.

4


5/7

Among apps analyzed in this

sample, the mean student

engagement rating is 82%.This high average rating

reects the fact that the apps

in this sample have been highly

curated with expected student

engagement as one criterion. A

random sample of educationalapps would likely have a lower

average engagement rating.

eSpark then calculates the

share of positive ratings to

total ratings to determine an

engagement metric for each

app in the curriculum. This

average rating is the dependent

variable in the regression

models, and app features are

the primary regressors. These

models also control for the

grade level and subject area of

the content.

Findings

The results from the prediction

equations are shown in Table

2. Among the curated set of

395 apps covering content for

grade levels PK-5, we show

the app characteristics thatare most strongly predictive

of student engagement. The

effect on student engagement

Attribute Defnition Effect on Student Engagement p-value

AnimationApp includes animation of graphicallycreated characters. +2.2% 0.029

Background music App has music in the background. +1.9% 0.021

Badges or rewardsStudents are rewarded for correct answerswith badges or rewards. +1.7% 0.087

Bare-bones interfaceInterface is black/white/green, low-budget, and does not boast stimulatinggraphics.

2.3% 0.039

Complex menu ofactivities

Menu of activities has many differentfunctions and may be difcult to navigate. 2.5% 0.055

Drill and kill Students practive the skill in a rote andrepetitive way. 2.5% 0.032

Graphics, colors, andsupplemental visuals

Graphics and supplemental visuals arecolorful and visually stimulating. +2.1% 0.003

SilentNo sound effects, music, or audio areincluded. 4.6% 0.001

Figure 4: Students are asked to rate each app in their personalized curriculum with

a thumbs up or a thumbs down button. The average app in this sample received

82 thumbs up for every 100 student ratings.

Table 2: The app characteristics that are most strongly predictive of student engagement among the curated set of 396 appscovering content for grade levels PK-5. 5


6/7

can be interpreted as the

contribution of a particular

attribute, holding all else equal,

to the average rating of apps

in this sample. For example,

among two otherwise similar

apps, an app with background

music is predicted to have a

mean student rating that is two

percentage points higher than

an app without music.

These results provide empirical

support for three main ndings.

First, students are very

sensitive to audio and visual

components of apps. Most

of the strongest predictors of

student engagement relate to

the app interface. Compared

to otherwise similar apps,

apps that have distinguishing

animation, music, or graphics

are predicted to have higher

student engagement ratings.

Apps that are silent or have

a bare-bones interface are

predicted to have signicantly

lower ratings than average

apps. This data suggests thatstudent engagement in iPad

apps is in large part driven by

whether apps can differentiate

the experience from a more

typical educational exercise.

Students gravitate toward rich

media and this is consistently

reected in their engagement

ratings.

Second, the data on studentengagement conrms some

common hypotheses on

pedagogical strategies.

Students respond positively

to badges and rewards. Apps

that reward students for correct

answers are associated with

an engagement rating that is

two percentage points higher.

Apps that follow a Drill and kill

strategy are associated with

ratings that are 2.5 percentage

points lower than average

apps. Although the teaching

medium changes when content

is delivered through an iPad

app, student responses to

pedagogical strategies remain

largely consistent. Student

motivation remains a central

component of the educational

experience.

Third, apps that offer a large

degree of choice reduce

engagement. Apps that offer a

complex menu of activities are

associated with signicantly

lower student engagement

ratings. This nding may

come as a surprise to many

educators. Several of the

most popular apps in the

App Store advertise extensive

functionalities. Educators often

gravitate to these omnibus

Figure 5: The app Fractions. Smart Pirate engages students with animated cartoons,

music, and stimulating graphics.

Figure 6: The app Counting Caterpillar, which teaches fundamental counting skills,

rewards performance with badges and rewards.6


7/7

apps because they offer so much content in

one package. That these comprehensive apps

are associated with reduced engagementsuggests that students may feel overwhelmed

by the upfront decisions required of these apps.

Clarity of use has a strong impact on student

engagement.

Whats Next

By linking qualitative characteristics of apps

to student engagement, eSpark has identied

empirical insights that were otherwise

unattainable at scale. This new knowledge hasled eSpark to rene its app curation process.

A deeper understanding of what kinds of apps

resonate with students has enabled eSpark

curriculum experts to make more sophisticated

design decisions. The newest eSpark curriculum

reects many of the insights described here.

eSpark continues to prioritize educational rigor

as the foundation of its curriculum, but now

curriculum experts are able to select content

with a much stronger understanding of how to

identify the most engaging apps.

eSpark hopes this data will reach further and

inform decisions among both educators and

app developers. When educators evaluate apps,

they would be wise to prioritize content that has

a rich interface, adheres to proven pedagogical

strategies, and prioritizes intuitiveness of use.

App developers can apply these lessons to

create more content that meets these best

practices.

The attributes discussed here merely scratch the

surface of the universe of app characteristics.

eSpark is continuing to develop its appdatabase by expanding attribute tagging in both

breadth and depth. Newly developed apps are

continually added to this catalogue, and apps in

the current eSpark curriculum are being tagged

for even more extensive attributes.

Looking forward, eSpark will use this attribute

database to develop a richer understanding

of how iPad apps impact student outcomes.

This study has explored the effects of app

characteristics on student engagement. Inthe future, eSpark will be able to link specic

app characteristics to impacts on student

achievement. This data will allow educators to

rene app curation even further, enabling them

to select the content that has been proven to

be most effective for improving student learning.

Looking forward even further, eSpark is excited

about a future where educators will be able to

match apps to students, at scale, based on

students interests, preferences, and motivations.

Figure 7: The apps Coin Math and iTooch offer many applications but require students to choose from a complex menu before

engaging with activities.

Maya LopuchData Scientist at eSpark Learning

Maya Lopuch leads research and analytics

at eSpark Learning. Prior to joining eSpark,

she was a researcher at the Harvard

Graduate School of Education, where

her work investigated how public schools

impact inequality and long term student

outcomes. Maya holds degrees in Economics and Public Policy

from Stanford University and the University of Chicago. She

believes in using data to inform and improve education. 7

Documents

Student Engagement in Educational Apps