Case studies of a robot-based game to shape interests and ... · non-traditional methods or multi-modal methods of in-struction can be more effective (Kroesbergen and Van Luit 2003;

Alfieri et al. International Journal of STEM Education (2015) 2:4 DOI 10.1186/s40594-015-0017-9

SHORT REPORT Open Access

Case studies of a robot-based game to shapeinterests and hone proportional reasoning skillsLouis Alfieri1*, Ross Higashi1, Robin Shoop2 and Christian D Schunn1

Abstract

Background: Robot-math is a term used to describe mathematics instruction centered on engineering, particularlyrobotics. This type of instruction seeks first to make the mathematics skills useful for robotics-centered challenges,and then to help students extend (transfer) those skills. A robot-math intervention was designed to target theproportional reasoning skills of sixth- through eighth-graders. Proportional reasoning lays the foundation for furtherprogress within mathematics. It is also necessary for success in a number of other domains (engineering, science,etc.). Furthermore, proportional reasoning is a life skill that helps with daily decision making, planning, etc. However,it is a skill that is complex and often difficult for students. Previous attempts to design similar robot-math activitieshave struggled to focus students’ attention on key mathematics concepts (in complex engineering domains), andto motivate students to use the math properly. The current intervention was designed with these challenges inmind. This intervention centers on a computer-based 3D game called Expedition Atlantis. It employs a game designthat focuses student attention on a specific proportional reasoning task: students calculate correct quantities ofwheel rotations to move the robot to desired locations. The software also offers individualized tutorials. Whole-classdiscussions around daily word problems promote further application of proportional reasoning outside the robotprogramming context. The 1-week intervention was implemented by three teachers at different schools withvarying levels of ability among students.

Results: Overall, within-participant comparisons revealed that the intervention was successful in improving thenumber of correct responses, the number of problems attempted, the proportions of correct responses, students’interest in robotics, and students’ valuing of mathematics within robotics from pre- to post-test. Further analysis ofteachers revealed that the two class sections of special education benefited most. Consideration was given to thequalities of the implementation that might have led to these enhancements.

Conclusions: The success of this intervention suggests that robot-math activities might be successful when focusedon a few target skills and when designed with individualized tutorials/prompts that motivate proper skills. Furtherinvestigations of student and implementation characteristics would help to refine these interventions further.

Keywords: Robot-math; Gaming to learn; Proportional reasoning; Special education

FindingsBackgroundRobotics is being taught in over 35,000 formal and infor-mal education settings in the USA (FIRST 2013; REC2013) and a number of educators and researchers havehighlighted the potential use of robotics lessons to rein-force students’ mathematical understanding (Benitti 2012;Vollstedt et al. 2007). One popular premise for doing so is

* Correspondence: [email protected] Research and Development Center, University of Pittsburgh, 3939O’Hara Street, Pittsburgh, PA 15260, USAFull list of author information is available at the end of the article

© 2015 Alfieri et al.; licensee Springer. This is anAttribution License (http://creativecommons.orin any medium, provided the original work is p

that students are more likely to recognize the math asrelevant because of its direct, applicable utility to a con-crete and contextualized task (Doppelt et al. 2008; Mubinet al. 2013). This instructional approach of teaching math-ematics within the context of robotics is commonly re-ferred to as robot math. Robot math’s cross-disciplinaryintegration of engineering, technology, and mathematicshighlights for students how these different disciplines ne-cessitate and facilitate each other. Such instruction standsin contrast to more traditional mathematics classes, whichmight prompt mathematical reasoning only for the sakeof understanding numerical relationships, and merely

Open Access article distributed under the terms of the Creative Commonsg/licenses/by/4.0), which permits unrestricted use, distribution, and reproductionroperly credited.

mailto:[email protected]

http://creativecommons.org/licenses/by/4.0

Table 1 Principles for MEA design

Principle Explanation

Reality principle Students can make sense of the situation basedon their experience.

Model construction The task creates a need for a mental model tobe constructed, modified, extended, or refined.

Model documentation Students are required to explicitly reveal howthey are thinking about the problem.

Self-evaluation The problem statement includes rubricsenabling students to judge for themselveswhether their solution is acceptable.

Generalizable model The model should not only work on the specificproblem but should be sharable and usable inother situations.

Simple prototype The problem should be as simple as possiblegiven the instructional goals.

Alfieri et al. International Journal of STEM Education (2015) 2:4 Page 2 of 13

describe it as occurring in a context. Although the currentpaper focuses exclusively on robot math, such integratedactivity-based instructional approaches might be appro-priate for STEM (science, technology, engineering, andmath) education more broadly as the lines between thesedisciplines increasingly disappear.When engaged in a robotics activity, students might

first encounter a mathematical problem within a con-crete context such as movement programming, and le-verage that context to begin sense-making at a higherlevel both faster and sooner. Through successful earlyencounters with problems of a mathematical nature,students could gain both experience and confidence inusing robot-contextualized math (those concepts/solutionscommon to robotics) within that single, well-definedcontext. They might then extend that understanding tomathematics more generally (by increasing the level ofabstraction of their mental models) and consequently,see how these solutions also connect to everyday math-related situations (generalize their solution strategiesand/or models in order to transfer them to other situa-tions; e.g., math classes, science classes, or even every-day life).However, successful mathematics learning can be dif-

ficult for students to achieve through robotics. Investi-gations of educational robotics (e.g., Silk and Schunn2008) have revealed that learners encounter difficultieslearning mathematics when too many topics are incor-porated into a single lesson. Moreover, subsequentstudies (e.g., Silk et al. 2010; Silk 2011) found that (a)the majority of teams in a middle-school robotics com-petition “guessed and checked” their ways through theprogramming portion of the task rather than use math-ematics, and (b) the success of teams that did attemptto use mathematical solutions was highly dependentupon the way in which the mathematics were integra-ted into the team’s overall problem-solving process.Finally, extended investigations of mathematics in ro-botics sometimes suffered from low engagement andthus no interest development in robotics or mathemat-ics, especially when the same basic robotics task neededto be revisited many times to deepen the mathematicalsolution.Robotics lessons designed to reinforce mathematical

understanding (i.e., robot math instructional methods)could therefore benefit from the following points of re-finement: (1) narrow the number/variety of mathema-tical concepts included within the lesson, (2) activelydiscourage the guess-and-check method, (3) improvethe use of mathematical solutions, and (4) provide a ro-botics context that is sufficiently varied to maintainengagement. The current study investigates whether arobot programming game using those strategies canproduce better student outcomes.

In particular, we explore whether a robot program-ming game is a platform that can facilitate robot mathby affording teachers with the opportunity to capturestudents’ attention and engage them in an activity that isenjoyable while also highlighting for them the utility ofmath both within the game and more generally. The re-search questions included (1) whether this approach iseffective in achieving math learning gains, and (2) whe-ther the approach also raises students’ interests in robo-tics, mathematics, or their awareness of the importanceof mathematics in robotics. The intervention was de-signed to be implemented within the kind of classroomtypical of robotics education settings: elective technologyclasses. And so, also of interest was the fidelity of teachers’implementation and whether gains would be found in thiselective type of classroom environment.

Supporting at-risk classrooms and teachers not certified inmathematicsImplementing robotics activities in elective technologyclasses to facilitate mathematics learning can be challen-ging because teachers in those classes are rarely certifiedin mathematics instruction. Therefore, we designed activ-ities that required little support from teachers by includingthe basic mathematics strategies in the instructions to stu-dents within the game. Furthermore, to allow for differen-tiated support for students of varying mathematics ability,we developed the robot math activities using the modeleliciting activity (MEA) design principles (see Table 1;Lesh et al. 2000; Reid and Floyd 2007). Activities designedwith these principles would challenge more advanced stu-dents while remaining approachable to weaker students.An additional benefit of robot math is its potential ap-

peal as an “alternative” method for classrooms of learnerswho are considered at-risk of not learning the mathemat-ics skills essential for testing and success using main-stream curriculum. For example, students grouped underthe umbrella of special education might struggle with


schoolwork for a variety of reasons (e.g., poor self-efficacy, poor reading ability, or attention disorders;Hampton and Mason 2003; Pastor and Reuben 2008;Zimmerman and Martinez-Pons 1990) and sometimesnon-traditional methods or multi-modal methods of in-struction can be more effective (Kroesbergen and VanLuit 2003; Moreno and Mayer 2007). Because of thepotential for robot math to engage students in a genu-inely motivating way by conveying useful mathematicsconcepts (grounded within robotics instead of de-scribed abstractly), these at-risk classrooms might havethe most to gain from such activities. The engagementof otherwise at-risk students paired with differentiatedstudent supports within the game (hints, tutorials, etc.)could maximize the potential for student success. Fur-thermore, the varying levels of ability within such class-rooms can be handled by the differentiated supportswithin the game, which include difficulty settings thatcan be adjusted to increase/decrease the complexity ofcalculations (e.g., the inclusion of decimal numbers).

The challenges of developing effective activitiesWhile there are many salient mathematics concepts thatcan be found within robotics activities typically experi-enced at the middle and high school levels, we focusedon proportional reasoning. Proportional reasoning is apervasive concept in mathematics (National ResearchCouncil 2001) that involves scale, rate, and conversionof units and has been identified as “the capstone of chil-dren’s elementary arithmetic and the cornerstone of allthat is to follow” (Lesh et al. 1988). Proportional reason-ing is also pervasive in robotics movement planning, aswheels and limbs typically move distances and anglesthat are proportional to motor commands.In Expedition Atlantis, we targeted this single key ma-

thematical concept for intense intervention and usedprinciples of MEAs to guide our material design (seeTable 1). Prior efforts with this approach had already re-vealed design issues that we attempted to avoid or miti-gate. For example, one such unit, robot synchronizeddancing (RSD), challenged students to program two ro-bots with different-sized wheels and wheel bases in orderto allow them to move (dance) in synchrony. A uniquesequence of dance movements programmed into onerobot had to be adapted to synchronize a second (differ-ently sized) robot. This overall task was divided into sub-tasks to scaffold the demands of the challenge (Apedoeet al. 2008): (1) synchronizing the distance traveled, (2)synchronizing robots’ speeds, and (3) synchronizing theirturns. As students worked, they were asked to createand revise a general technique that would allow for thesynchronization of more robots of different sizes (i.e., ageneralizable model of the solution). Additionally, stu-dents solved math-focused abstraction bridges (Brown

and Clement 1989) that prompted students to solve robotmath word problems. These problems were initiallyanalogous to the calculations of distance and speed re-quired for the challenge, but progressed toward everydayproportional reasoning scenarios to prompt students togeneralize the proportional reasoning skills that theyhad developed within the robot challenge to more tra-ditional situations, thereby developing a more generalunderstanding.The first version of RSD fell short in implementation

because students spent too much time on developingthe base dance. Instead of tackling the mathematics in-volved in the challenge, initial lessons had to teach thenecessary programming language commands, and stu-dents spent too much time working on the esthetics ofthe dance. Initial revisions to the module mitigated bothissues by providing the base dance to students, but en-countered another layer of implementation challengesrevolving around the inaccuracies and inconsistency ofrobotic systems in physical operation. In particular, thereal-world noisiness of physical robotics created a dis-traction: the teacher needed to facilitate additional dis-cussions about why averaging across multiple trials wererequired, why precise measurement was necessary, etc.(Silk 2011).As a result of these challenges, the current interven-

tion uses a game-like robot simulator called ExpeditionAtlantis that operates entirely in a “virtual world,” thuseliminating the mathematical challenges of stochasticperformance and measurement error. It also limits thescope of programming to the selection of the most ma-thematically vital parameter: how many wheel rotationsthe robot needs to move or turn to achieve the desiredmovement outcome. These differences reduce the bur-den on teachers and students to first learn programmingbefore learning the math, as well as highlight propor-tional reasoning without requiring attention to statisticsand measurement concepts.Furthermore, to reduce the chances of guess-and-check

methods (what students do when they are avoiding themathematics), Expedition Atlantis includes automatictutorial prompts after the player repeatedly answers in-correctly. With these built-in tutorials and prompts forstudents to develop their skills independently, teachersare not expected to do much more than guide discus-sions of the solutions to accompanying word problems(abstraction bridges). The abstraction bridges writtenfor the RSD intervention (Silk 2011) were adapted tothe challenges within Expedition Atlantis. The content ofthe abstraction bridge problems initially center aroundthe Expedition Atlantis game, then gradually shift toother contexts. As before, the intent is for studentsto develop proportional reasoning skills by first creat-ing models of the proportional relationships within the


game, then extending those models to everyday situa-tions over time.On the basis of prior results and the additional issues

addressed in Expedition Atlantis, we hypothesized thatwe would find increases in proportional reasoning testscores from pre- to post-test, as well as increases inmeasures of students’ interests and values pertaining tomathematics, robotics, and the use of mathematics inrobotics.

MethodsParticipantsThe intervention was implemented by three teachers atdifferent schools within urban/suburban districts in thenortheastern USA; 116 students participated. Teacherswere financially compensated for their time, which in-cluded a 2-h professional development session beforeimplementation was scheduled to begin. Teachers at allthree schools implemented the intervention in two oftheir class sections ranging from grade 6 through 8.Table 2 lists the numbers of students and their mathem-atics levels by section.Schools were selected in order to explore the impact

of the intervention across different classroom contexts.Two of the schools were public, one private. Both classsections at school A (the first public school) were la-beled as special education. These students were consid-ered to be at-risk of not passing state standardized tests.At school C (the second public school), one of the classsections was labeled as accelerated and the other sectionwas at a standard level. Because school B (the privateschool) included a mix of grade levels (6 through 8)within its sections, ability levels varied within each sec-tion but all were around average for the students’ re-spective grade levels. School B was in an urban districtas compared to the more suburban settings of thepublic schools (A and C). The teachers at schools Aand C were not certified in mathematics but the teacherat school B was certified in secondary (grades 7–12)mathematics.

Table 2 Sample of students across three locations

Context 6th graders 7th graders 8th graders

School A (public)

Special education section 1 15

Special education section 2 15

School B (private)

Mixed section 1 10 7 6

Mixed section 2 8 8 5

School C (public)

Average section 22

Accelerated section 20

At the end of a pre-test, students were asked if/whenthey would complete an algebra course and to rate theirlevels of experience with robots, among other questionson a survey of prior experience. Survey responses indi-cated that the majority of students (n = 66) planned totake a full algebra course in the eighth grade (includingall 11 eighth-grade students, who were already doingso). The 20 students within the accelerated sixth-grademath class would take an algebra course in seventh-grade and the 30 students within the special educationclasses would take such a course in ninth grade. Themajority of students (n = 99) had little or no experienceworking with robots. Only four students reported hav-ing extensive experience with robots and removingthese students from analyses did not significantly alterthe results.

MaterialsCurriculum At the center of the intervention was aninteractive simulator game (Expedition Atlantis) requir-ing students to utilize their proportional reasoning skillsin order to program a robot to navigate through a 3Daquatic environment. Figure 1 provides an overview ofthe entire game (its display, example problems fromeach chapter, etc.).Students interact with the game by filling in para-

meters to control the robot’s motors during a series ofmaneuvers. These parameters are expressed in units in-ternal to the robot, such as motor/wheel rotations. How-ever, each parameter exposed in this way is proportionalto a tangible in-game goal quantity, such as distance therobot travels. Students are given a goal in each level ofthe game that (implicitly) requires them to relate thetwo. For instance, because wheel rotations are propor-tional to the distance the robot travels, one level tasksstudents with making the robot move a specified dis-tance by determining the corresponding number ofwheel rotations to make it do so. A correct answer ad-vances the student in the story, while an incorrect answerprompts the software to begin offering progressivelystronger hints, eventually directing students through anadditional series of activities that use a building-up ap-proach to illustrate the physical relationship betweenthe number of wheel rotations and the distance traveled,using the values from the student’s current problem.The game contains five substantively different scenar-

ios, spread over four chapters, each raising an additionalcomplication for the robot. The first level involves onlyforward movement; the second introduces turning; thethird scenario combines moving and turning; the fourthintroduces physical changes to the robot that alter theunderlying rate values (e.g., a larger wheel goes fartherin the same number of rotations); and the fifth sce-nario requires students to plot out their own movement

Figure 1 Chapters within Expedition Atlantis.


sequences in order to navigate a maze-like environment,and subsequently solve the individual movement-valueproblems that the sequence generates. See Figure 1 for an-other description of individual chapters, explanations ofthe display, and example problems students had to solve.When used in a classroom setting, teachers could le-

verage each new scenario as an opportunity to examine,discuss, and reason about the nature of the proportionalrelationships and how best to describe and apply themin the game context.

Each school was provided with access to the programfor installation on student computers. In addition to a 2-hprofessional development session prior to implementa-tion, teachers received binders that explained the purposeof the game and its accompanying activities.Each day’s activities were accompanied by worksheets

that prompted students to complete proportional rea-soning practice problems called Abstraction Bridges.Our intention was to have these problems help studentsto generalize the mathematical skills and knowledge


gained within the game. On the first day, students com-pleted problems that were very similar to calculationsneeded within the game (e.g., moving and turning therobot; near transfer). On each successive day, problemson the worksheets became increasingly dissimilar intheme; then in form, from the calculations within thegame until on the last day(s), problems were completelyunrelated to robot movement (e.g., plow sizing, paintmixing, etc.; i.e., far transfer). See Table 3 for examplesof both near and far transfer practice problems.The game allowed teachers to decide which difficulty

levels were most appropriate for students. This settinginfluenced the difficulty of the calculations (e.g., thepresence or absence of decimal numbers), but not theorder of robotics topics (e.g., straight movement was al-ways followed by turning). Similarly, difficulty settingsdid eventually influence areas directly relevant to thetarget topic of proportional reasoning (e.g., the inclusionof non-unit base rates; 2 rotations = 3 m) at higher set-tings. Figure 1 provides prompts at different difficultylevels for comparison. Teachers initially set all students’difficulty to a common level based on the mathematicslevel of the class, and then directed individual studentsto increase or decrease their individual difficulty levelsbased on their rate of progress through the game.

Proportional reasoning test There were two versionsof the test that were administered roughly equally as ei-ther pre- or post-tests within classrooms (adapted fromSilk 2011; Weaver and Junker 2004). The two versionswere comparable in content (i.e., mostly far transfer, pro-portional reasoning problems some of which could besolved without complex mathematical calculations) andcontained the same number of question prompts. See

Table 3 Abstraction bridges

When encountered Name of problem Problem

Following day 1 Robot movements Your robot moves 10 m in 4

How many wheel rotations do

Following day 2 A new robot You just purchased a new robthe robot, it turns 5°.

How much will your robot turn

Following day 3 Going on a trip You are traveling to visit a friefill up. You will use 54 gal of

How much will you need to sp

Following day 4 Paint mixing When we arrived at the robocustom colors. A gallon of redcolor that we want.

If the job is going to take 24 g

Density of tire spikes We have to travel across ice wwheel doesn’t slide. Our ice ttires have more spikes but are

If each type moves 12 m, whic

Table 4 for example questions. Students received oneversion as a pre-test and the other as a post-test. Priorwork using this assessment failed to find reliable pre/post improvements with similar populations in severalcontrol groups and in some of the robotics-related in-terventions (Silk 2011). Therefore, it is unlikely thatany gains from pre to post in this study would simplybe the result of increased familiarity with the questions,a Hawthorne effect (i.e., gains simply from experiencingnovel instructional approaches), or typical ongoing math-ematics instruction during this time period. Because thecurrent study could not investigate students’ transfer ofpotentially gained math skills in other subjects’ classes, thequestions on pre- and post-tests were chosen to be out-side of robotics and in contexts more like everyday life.

Prior experience survey At pre-test, students were alsoasked to answer questions about their grade level,algebra-taking plans, and familiarity with robots.

Motivation survey At both testing times, students ratedtheir personal interests in and values of mathematics androbotics on a five-point scale from NO! (strongly disagree)to YES! (strongly agree). Students rated statements aboutmathematics interest (four items; Cronbach’s alpha: pre =.74; post = .82), robotics interest (four items; Cronbach’salpha: pre = .73; post = .76), and the value of math in ro-botics (four items; Cronbach’s alpha: pre = .66; post = .80).In prior work using these scales (for a summary, see Silk2011), robotics interventions either tended to increasemotivation or proportional reasoning but not both. Again,that a number of prior robotics interventions showed nopre/post gains on these measures rules out gains fromsimple test-retest effects or Hawthorne effects.

wheel rotations.

es it take to move 30 m?

ot! The instructions say that for every 8 motor rotations programmed in

if it uses 56 motor rotations?

nd in another state. Your car has an 18 gal gas tank and costs $48.00 togasoline in your travels.

end on gasoline?

t garage, we were informed that we could customize our robots with, 2 gal of white, and a gallon of blue paint will give us one particular

al of paint total, how many gallons of each color will we need?

ith our ice tires. Ice tires have little spikes that dig into the ice so theires are 6 m but Goodtire 3000 makes much smaller ice tires. Our iceless dense.

h one will create more spike marks on the path?

Table 4 Example questions from the proportional reasoning test

Version 1 Version 2

Question Question

1) There are 36 poker chips, 12 blue (B) and 24 red (R). 6) Frances has an enlarging machine that can enlarge or reducea photograph to any size while keeping the same shape. If anoriginal photograph is 2 in wide by 2.4 in long and Franceswants to enlarge it to be 5 in wide, how long will it be?

Arrangement 1 Arrangement 2

4B 4B 4B 3B 3B 3B 3B

8R 8R 8R 6R 6R 6R 6R

What changed between the first and second arrangements?

What did not change?

Show another arrangement of the poker chips that preserves the same relationship.

5) John is 15 years old. John’s father is now three times as old as John is. 11) It takes Vanessa 9 h to paint 15 chairs. How long will it takeher to paint 20 chairs?

How old is John’s father now?

When John is 20, how old will his father be?

14) The ratio of 5 to 4 is the same as: 13) Which statement is correct?

a) The ratio of 6 to 5 a) 2/5 = 1/4 = 3/6

b) The ratio of 5/4 to 1 b) 2/5 = 4/25 = 16/625

c) The ratio of 10 to 8 c) 2/5 = 6/15 = 4/10

d) b and c


Design and procedurePre- and post-tests were administered by the teachers inthe presence of the experimenter on the days immedi-ately preceding and following the roughly 5-day inter-vention. An experimenter was present in all classroomsthroughout implementation to ensure the materials werebeing implemented, assess student reactions, and handletechnological obstacles (e.g., game freezing). To maintainthe authenticity of the implementation, the experimenterimposed few restrictions or recommendations. Teachersled abstraction bridge discussions as they would in theabsence of the experimenter and students worked indi-vidually or in pairs as they traditionally would. The ex-perimenter met with the teacher before and/or aftereach class to discuss the day’s activities, the subsequentday’s activities, and any other concerns that the teacherhad. However, teachers raised few questions as to howto proceed with the intervention. Most discussions withteachers centered around students’ progress and promp-ted feedback from teachers about the intervention moregenerally. Whereas stricter guidance would have increasedthe amount of experimental control, it potentially couldhave made the class seem unusual to students and conse-quently suspect. For these reasons, it seemed best for thisfirst study of the unit’s implementation to allow thevariations inherent to different teaching styles andclassroom environments, which would further allow usto explore preliminarily which variations were mosteffective.Before the first day of implementation, students were

pre-tested. The intervention was then implemented across

subsequent days; students engaged in game play and thenreceived the day’s practice problems to complete either atthe end of class or for homework. Except for the first dayof game play in which students immediately began withgame activities, all other days began with the teacherreviewing the practice problems (abstraction bridges)with the class in the form of a class discussion andmodeling (either by classmates or the teacher) of thecorrect solution(s). Reviewing solutions to a problemrequired about 5 to 10 min. After the review session,game play resumed at whichever chapter the studenthad left off. Following the intervention, post-testswere administered.It warrants noting that the current investigation was

not designed to directly compare outcomes betweenschools. Instead, it was designed to explore the variety ofways that teachers implemented the intervention andthe effects of that implementation on learning outcomes.That is, this investigation examines effects within threepurposely-diverse sites as case studies. The examinationof these three implementations provided insight into theefficacy of the teacher’s guide, the fidelity of implemen-tation, and the efficacy of the intervention following thatimplementation.

Results and discussionPre- and post-tests were scored by the experimenter,providing partial points for correct reasoning even if cal-culated incorrectly (e.g., rounding error). Two test scoreswere created: raw scores of the number of correct re-sponses (out of 17), and proportions of answers correct


out of the number attempted by each student becausestudents sometimes skipped questions. Changes in stu-dent interest in mathematics and robotics, and recognizedvalue of math in robotics were also scored to investigatewhether participation in the intervention changed stu-dents’ opinions.The proportional gains from pre- to post-test were

subjected to an ANCOVA that included ability level as afixed effect and pre-test number correct and pre-testnumber attempted as covariates. Because ability level, lo-cation/teacher, and grade level were all highly contingent(contingency coefficients of 0.8, p < .001), we chose tofocus analyses only on ability levels. The ANCOVA’scorrected model for proportional gains was significant,F(4, 100) = 9.07, p < .001 with a marginal effect of classcategory, F(2, 100) = 2.92, p = .06. However, there stillseemed to be reason to suspect that there were diffe-rences in implementation at different locations - par-ticularly when comparing students of traditional abilitylevels. Therefore, the data were analyzed with pre/postpaired-sample t-tests, first including all locations, andthen, follow-up analyses for each location separately.This was done for both sum scores and the proportionscorrect, as well as mathematics-, robotics-, and value-related self-ratingsa. Effect sizes to capture pre/postchange were then calculated by using groups’ pre- andpost-test measures’ means, standard deviations, andcorrelations between measures (in order to adjust theeffect-size for a within-subjects design).

All three schoolsOverall, we found that across locations/teachers, theintervention improved scores on proportional reasoning(both sum scores and proportions correct). See Table 5for a presentation of all measures, and Figure 2 for effectsizes. We also see that there is a marginal (p = .061) in-crease in the number of prompts attempted by studentsfrom pre- to post-test. Thus, the large change in overallnumber correct reflects changes in both numberattempted and accuracy within those attempted.We also see significant increases in student robotics inter-

est and students’ values of mathematics for use in roboticsfrom pre- to post-test. There was not a significant increasein students’ general ratings of mathematics interest, againruling out a Hawthorne effect in which students broadlyshow change as a result of exposure to a new environment.To test whether the mathematical gains depended

upon initial attitudes toward mathematics or robotics,correlations were computed between pre-test interest orvalues ratings (for mathematics, robotics, mathematicsin robotics, or the combined average of all three) andgains in proportional reasoning. None of these correla-tions was statistically significant (Pearson r = −.09, −.11,.03, and −.09, respectively).

Individual schoolsThe patterns of results across our three locations indi-cated some variability in effects but generally, patternswere consistent. As can be seen in Table 5, marginal ornon-significant differences were found between pre- andpost-test scores at the schools B and C but the specialeducation classrooms at school A did show gains largeenough to be statistically significant on their own. Onaverage, classes at all locations increased in their inter-ests in robotics, and classes at two of the three locationsincreased in their value of mathematics for use in robot-ics. After only several days of game play, these are con-siderable gains. Furthermore, every setting saw gains onat least two measures.

Observations of classroom environmentsAll three teachers maintained high levels of implementa-tion fidelity across days. None of the three teachers devi-ated from the pace or purposes of the materials as theyhad been prepared. Students worked largely independ-ently at individual computers at schools A and C butpaired up to play the game at school B. Across all threelocations, few if any students asked for help in playingthe game. Teachers and the researcher circled to tacklemore technical problems (computers freeze, graphics mal-function, etc.), which were the few times students re-quested assistance. Teachers had little to do but walkaround and monitor student progress while studentsplayed the game. Their part in this intervention was mainlyto lead discussions of the abstraction bridge problems.In this capacity, the three teachers behaved somewhat

differently. As the abstraction bridge discussions wereunscripted, teachers led students through solution stra-tegies as they thought appropriate.Discussions of the abstraction bridges at schools B and

C included presentations of multiple solution strategies(e.g., the base rate or segment strategy, the T chart strat-egy described below, or the scale factor strategy de-scribed belowb) by students who displayed and explainedtheir reasoning behind the solution. Teacher B (i.e., atschool B) decided who would present her/his strategyafter asking students what their answers were. A firststudent would provide an answer and others wouldagree or disagree by a show of hands. The teacher wouldthen typically ask the first (or multiple students) to showhow (s)he solved the problem. Consequently, all strat-egies led to similar answers (some rounding variations)and all were generally correct - as selected by the teacher.In comparison, teacher C chose students either becausethey volunteered or because he wanted to challenge spe-cific students. This led to student presentations of bothcorrect and incorrect strategies and answers, which theteacher then labeled as correct or incorrect after theirpresentation. Consequently, more students presented

Table 5 Pre/post gains in proportional reasoning and interests (statistically significant gains in italic)

Measure Pre-test Post-test

M SD M SD t statement

Sum score 6.74 3.50 7.97 3.64 t(104) = 4.20, p < .00

School A 6.14 3.00 9.71 2.09 t(27) = 6.80, p < .00

School B 8.89 3.62 9.50 3.73 t(37) = 1.31, p = .20

School C 5.08 2.59 5.23 2.70 t(38) = 0.41, p = .69

Proportion correct .56 .22 .62 .21 t(104) = 2.85, p < .01

School A .52 .19 .66 .15 t(27) = 3.83, p < .01

School B .68 .20 .68 .18 t(37) = 0.00, p = 1.0

School C .46 .20 .53 .25 t(38) = 1.44, p = .16

Prompts attempted 12.27 4.08 13.08 4.38 t(104) = 1.90, p = .06

School A 11.93 3.62 15.00 2.98 t(27) = 4.57, p < .00

School B 13.37 4.09 14.11 3.93 t(37) = 1.37, p = .18

School C 11.44 4.24 10.69 4.65 t(38) = −0.91, p = .37

Student interest in mathematics 2.20 0.96 2.21 1.09 t(104) = 0.16, p = .87

School A 1.97 1.05 2.06 1.20 t(27) = 0.86, p = .41

School B 2.17 0.77 2.29 0.94 t(37) = 0.96, p = .35

School C 2.36 1.03 2.24 1.14 t(38) = −0.93, p = .36

Student interest in robotics 2.48 0.74 2.84 0.82 t(104) = 4.84, p < .00

School A 2.29 0.58 2.57 0.78 t(27) = 2.40, p = .02

School B 2.63 0.80 2.96 0.72 t(37) = 2.96, p = .01

School C 2.48 0.75 2.90 0.91 t(38) = 3.00, p < .01

Student values math in robotics 2.46 0.64 2.81 0.83 t(104) = 4.29, p < .00

School A 2.28 0.68 2.52 0.81 t(27) = 1.50, p = .15

School B 2.49 0.65 2.95 0.72 t(37) = 3.55, p < .01

School C 2.53 0.61 2.83 0.91 t(38) = 2.24, p = .03


different strategies at school C (both correct and incor-rect) than did students at school B. Furthermore, the timespent on discussions of these problems were generally lon-ger at school C than at the other schools (precise timemeasurements unavailable) because a number of incorrectresponses needed to be explained before a correct solutionwas presented. This also slightly decreased the amount oftime spent playing the game.One of the strategies presented at school C included

an approach not seen at the other two schools: the Tchart. The T chart is a method of increasing two num-bers incrementally through addition on both sides (e.g.,doubling 18 gallons and $63.00 until $63.00 is close to$192.50 in order to figure out how many gallons of gaswere purchased for $192.50). Answers following fromthis strategy are approximate at best. In comparison toother solution strategies, this approach seemed ineffi-cient but was observed in both the traditional and accel-erated classrooms at school C.Discussions at school A surrounded the use of only a

single strategy - scale factor. As at the other schools,

students here were asked to present their solutions ei-ther because they volunteered or because the teacher en-couraged a student personally. On the first day ofabstraction bridge discussions, a student identified andexplained her scale factor strategy and the other studentswere quick to recognize it from a previous class. Subse-quently, all students adapted the same scale factor strategy- or a closely similar variant. This allowed discussions tofocus on (1) the relationships between numbers (e.g., la-bels such as meters and rotations and the relationship be-tween number of rotations and distance traveled inmeters), (2) what each number represented at each stageof the solution (e.g., the distance per single rotation,the cost per gallon of gas, etc. - numbers calculated inthe process of reaching the final answer), and (3) thedetails of working through this mathematical procedure(e.g., rounding, multiplying without a calculator, etc.).These three components were present in all discussionsat school A but were less commonly present in discus-sions at the other two schools. Later problems eliciteddiscussions as to how to apply the scale factor strategy

Figure 2 The effect sizes for skill development and interest.


to these more complex “life math” problems - as re-ferred to by teacher A.Thus, the particular gains observed in school A might

not be due to student characteristics (special education)alone, but also to teacher A’s organization of abstractionbridge discussions. It may be that students benefit mostwhen the number of strategies presented to solve prob-lems is kept low and when only correct strategies arehighlighted. Furthermore, teacher A elicited explanationsof these strategies in particular ways: asking students whateach number represents, the purpose of each mathematiccalculation, etc. However, given the various differences be-tween school contexts, no strong inferences can be madeabout the effects of the teacher strategies.

ConclusionsOverall, the 1-week intervention increased students’ pro-portional reasoning skills from pre- to post-test (particu-larly at school A), as well as their interest in robotics (allthree schools) and their value of mathematics for usewithin robotics (particularly at schools B and C) as in-ferred from within-participant comparisons. In past stud-ies using physical robots without the additional structuresprovided by the Expedition Atlantis game, limited math-ematics gain was generally found. In the one prior case inwhich significant gains were found using the physical ro-bots, the students were from an advanced private schooland the instructor was from the research team, and there-fore able to provide many complex supports for the

students that would be difficult for traditional technologyeducators to provide effectively. Finally, the prior casewith mathematics gains did not show gains in interest inmathematics or robotics (Silk 2011). Thus, the currentoutcomes were generally more positive, especially giventhe implementation in challenging contexts without trad-itional technology instructors.We believe these improved outcomes to be the result of

four major features of the intervention. The first featurewas a focus on a single mathematics topic: proportionalreasoning. This allowed students to concentrate on thisskill instead of attempting to learn all of the mathematicspotentially involved within robotics. The second was asimplification of how the robot needed to be programmed.Instead of requiring students to learn the syntax of pro-gramming, they could directly enter their calculations forthe number of wheel rotations required for a given destin-ation. The third feature was a curbing of students’ propen-sity for guess-and-check approaches. After a few incorrectguesses, students’ attention was drawn to particular piecesof information on screen (e.g., the distance for a givennumber of rotations) needed to solve the calculation cor-rectly. After a few more incorrect guesses, students wereassisted by a pop-up tutorial within the game that demon-strated how to think about the problem, how to imaginethe distances, and how to calculate the number of rota-tions needed. Although most students who encounteredthe tutorial were grateful for the personal assistance, somestudents were frustrated because they had reached the


tutorial when their guess-and-check approaches were in-sufficient. Nevertheless, these students were observed tobe less likely to guess in the future. Lastly, the fourth fea-ture was the inclusion of abstraction bridge word prob-lems that facilitated a generalized application ofproportional reasoning to other domains, thereby prepar-ing students to use this reasoning for analogous problemsand/or in daily life. Thus, these initial findings would pre-dict that robot-math activities designed with similar fea-tures would also likely be successful.Again, the findings reported here are case studies of

three locations with conclusions drawn from within-participant comparisons. Further investigations are ne-cessary to tease apart these features of the intervention -perhaps including a comparison to a meaningful baselinecontrol group (no instruction, practice only, traditionalinstruction, etc.).

Supporting teachers not certified in mathematics andat-risk classroomsWhen schools were examined individually, the greatestgains were found at school A where both class sectionshad been classified as special education (at-risk) and theteacher was not certified in mathematics. We had hopedof course that our intervention would improve scores ofall students but were particularly interested in its effectswithin special education classrooms because many ofthese students are considered to be at-risk of failingbenchmark tests in mathematics. We hypothesized thatthis “alternative” method of teaching proportional rea-soning may be successful because of its non-traditional,multi-modal approach that makes the math skills ac-quired transparently useful (within the game immedi-ately, and later within daily situations as discovered indiscussions of abstraction bridges; Kroesbergen and VanLuit 2003; Moreno and Mayer 2007). Furthermore, thestudent supports present within the game (highlightingof important information followed by tutorials) could ef-fectively scaffold the calculations to make them access-ible to students at varying levels of ability. And indeed,students at school A benefited considerably from theintervention.However, these initial findings require further investi-

gation as to the influences of student and/or teachercharacteristics. It is difficult to distinguish the two withthe current sample because only teacher A was responsi-ble for at-risk classes. Moreover, teacher A was exceptionalin fostering students’ understanding of the abstractionbridges. After the first student who presented a solutionreminded her classmates of the scale factor strategy byusing it successfully within the problem, all studentsadapted that strategy to meet the demands of subsequentproblems. The scale factor solution strategy had been in-troduced in a previous class and so all students were

familiar with it. This allowed teacher A to focus on themathematics content handled by the strategy instead of onthe selection of a proper strategy. She mediated discus-sions by focusing on the relationships between numbers(through the use of labels such as meters and rotations -e.g., the relationship between numbers of rotations anddistance traveled in meters), on what each number repre-sented at each stage of the solution (e.g., the distance persingle rotation, the cost per gallon of gas, etc. - numberscalculated in the process of reaching the final answer),and on the details of working through this mathematicalprocedure (rounding, multiplying without a calculator,etc.). Thus, teacher A was not limited to discussions as towhich mathematical procedure was best because thatprocedure was already a more-or-less routinized practiceof students. She could instead explain the mathematicalunderpinnings of that practice as it related to the prob-lems, thereby focusing on the content of proportionalreasoning skills. Again, further investigations could dis-aggregate the effects of the intervention, the teacher, andother factors to better understand each.The game at the center of the intervention seems well

designed for its purpose and our results suggest focusingfurther development on the activities surrounding thegame. If after some further research teacher A’s approachto abstraction bridge discussions is found to be success-ful with other students at other locations, professionaldevelopment training and the teacher manual might berevised to help teachers adopt such an approach. Whereasit seems unlikely that students will all have a shared previ-ous experience of using a particular solution strategy (aswas the case at school A), teachers can nonetheless ap-proach discussions in much the same way - by highlight-ing the relationships between the numbers, requiringstudents to explain what each number represents at eachstage of the solution, and fostering proper mathematicalhabits in rounding, multiplying, etc. This might seem likean intuitive approach to discussing mathematics problemsbut our evidence suggests otherwise. Perhaps the morecommon intuition is to present as many solution strat-egies as possible and to allow students to decide whichthey prefer of those decidedly correct. This is not to implythat allowing students to present incorrect solutionsshould be considered unproductive. In fact, teachersmight find them to be teachable moments that canhighlight how the strategy is unsuccessful for the prob-lem at hand - not only in regard to the final calculationbut also to the incorrect/meaningless numbers gener-ated at each stage of the solution. Whereas scriptingteachers’ contributions seems like a rigid approach,explaining to teachers what we have found to work(whether within their manuals and/or as part of profes-sional development) could foster these behaviors intheir own implementations.


In summary, this preliminary study suggests that thedesign of the Expedition Atlantis intervention, whichincluded four main features that were informed by thefindings in previous robot-math interventions (e.g.,RSD), was successful in fostering both students’ propor-tional reasoning skills and personal interests/values asdetermined by within-participant comparisons. How-ever, further investigations of student and teacher char-acteristics seem warranted to determine the influences/interactions of both in order to be able to maximize thepotentials of robot-math activities.The current effort represents but one type of inte-

grated STEM instruction, namely technology and engin-eering used to teach mathematics. Within that type ofSTEM instruction, it provides a model for how technol-ogy and engineering could be used more often to teachmathematics: focusing on one kind of mathematics, de-veloping model eliciting activities centered on thatmathematics, using rich game-like scenarios to main-tain engagement, and abstraction bridges to generalizethe learning. Similar methods could be used in otherforms of STEM education - engaging students withapplication-focused, real-world activities that do not cen-ter the to-be-learned skills as the ultimate goals but in-stead necessitate and motivate (particular, repeatedlypracticed) skills as tools.

EndnotesaBecause some students scored as outliers on a number

of measures (number correct on the pre-test, self-ratingof familiarity with robots, pre-test ratings of interests orvalue), those students were removed from analyses toassess their influence. This was true of 33 of our 116students but when removed, analyses revealed the samepattern of results. Consequently, they remained included.This is true both of the previously presented ANCOVAand the subsequently presented t-tests.

bThe base rate strategy label is one added by the authors.Students most often did not provide a name for theirstrategy - unless otherwise noted in the case of T chartsand scale factors.

Competing interestsExpedition Atlantis was created by a team from Robot Virtual Worlds, whichis part of RoboMatter Incorporated. Authors Shoop and Higashi are/werepart of the company at the time of study. Author Alfieri was unaffiliated withthe company at the time of the study but has since accepted a positionthere.

Authors’ contributionsLA helped to revise the teacher manual, provide professional developmentto teachers, execute the study (was the experimenter in attendance withinclassrooms), collect and analyze the data, and write the majority of themanuscript. RH was responsible for the design of the intervention and itsmaterials, the provision of professional development, interpreting analyses,and revising the manuscript critically. RS was responsible for offering thispreliminary study and in the design of the intervention. CDS acted as ananchor author having experience with the Silk studies conducted prior and

contributed critically to the design of the study, the interpretation offindings, and revisions of the manuscript. All authors read and approved thefinal manuscript.

Authors’ informationNote: The research reported herein was supported by award #FA8750-10-2-0165 from the Air Force Research Laboratory (AFRL) as part of the DefenseAdvanced Research Projects Agency’s (DARPA) ENGAGE initiative, RobinShoop, PI. The opinions are those of the authors and do not represent thepolicies of the funding agency.

Author details1Learning Research and Development Center, University of Pittsburgh, 3939O’Hara Street, Pittsburgh, PA 15260, USA. 2National Robotics EngineeringCenter, Carnegie Mellon University, 10 40th Street, Pittsburgh, PA 15201, USA.

Received: 3 June 2014 Accepted: 29 January 2015

ReferencesApedoe, XS, Reynolds, B, Ellefson, MR, & Schunn, CD. (2008). Bringing

engineering design into high school science classrooms: the heating/cooling unit. Journal of Science Education and Technology, 17(5), 454–465.doi:10.1007/s10956-008-9114-6.

Benitti, FBV. (2012). Exploring the educational potential of robotics in schools: asystematic review. Computers & Education, 58(3), 978–988.

Brown, DE, & Clement, J. (1989). Overcoming misconceptions via analogicalreasoning: abstract transfer versus explanatory model construction.Instructional Science, 18, 237–261.

Doppelt, Y, Mehalik, MM, Schunn, CD, Silk, E, & Krysinski, D. (2008). Engagementand achievements: a case study of design-based learning in a sciencecontext. Journal of Technology Education, 19(2), 21–38.

FIRST. (2013). The science of celebration: 2013 annual impact report. Retrieved fromhttp://www.usfirst.org/annual-report.

Hampton, NZ, & Mason, E. (2003). Learning disabilities, gender, sources ofefficacy, self-efficacy beliefs, and academic achievement in high schoolstudents. Journal of School Psychology, 41, 101–112. doi:10.1016/S0022-4405(03)00028-1.

Kroesbergen, EH, & Van Luit, JEH. (2003). Mathematics interventions for childrenwith special education needs: a meta-analysis. Remedial and SpecialEducation, 24(2), 97–114. doi:10.1177/07419325030240020501.

Lesh, R, Hoover, M, Hole, B, Kelly, A, & Post, T. (2000). Principles for developingthought-revealing activities for students and teachers. In AE Kelly & RA Lesh(Eds.), Handbook of Research Design in Mathematics and Science Education(pp. 591–646). Mahwah, NJ: Lawrence Erlbaum Associates.

Lesh, R, Post, T, & Behr, M. (1988). Proportional reasoning. In J Hiebert & M Behr(Eds.), Number Concepts and Operation in the Middle Grades (Vol. 2,pp. 93–118). Reston, VA: National Council of Teachers of Mathematics.

Moreno, R, & Mayer, R. (2007). Interactive multimodal learning environments.Educational Psychology Review, 19(3), 309–326. doi:10.1007/s10648-007-9047-2.

Mubin, O., Stevens, C. J., Shahid, S., Al Mahmud, A., & Dong, J. J. (2013). A reviewof the applicability of robots in education. Journal of Technology inEducation and Learning, 1. Retrieved from http://roila.org/wp-content/uploads/2013/07/209-0015.pdf

National Research Council (2001). Adding it up: Helping children learnmathematics. In J. Kilpatrick, J. Swafford, & F. Bradford (Eds.), MathematicsLearning Study. Center for Education, Division of Behavioral and SocialSciences and Education. Washington, DC: Nation Academy Press

Pastor, PN, & Reuben, CA. (2008). Diagnosed attention deficit hyperactivitydisorder and learning disability: United States, 2004–2006. Vital HealthStatistics, 10(237), 1–15.

REC. (2013). Inspire: Annual report 2013. Retrieved from www.roboticseducation.org/about/.

Reid, K, & Floyd, C (2007). The tsunami model eliciting activity: Implementationand assessment of an interdisciplinary activity in a pre-engineering course.Proceedings of the 2007 American Society of Engineering Education NationalConference. Retrieved from www.asee.org/search/proceedings?search=ses-sion_title%3A%22Engineering+in+Middle+Schools%22+AND+conference%3A%222007+Annual+Conference+%26+Exposition%22

http://www.usfirst.org/annual-report

http://roila.org/wp-content/uploads/2013/07/209-0015.pdf

http://roila.org/wp-content/uploads/2013/07/209-0015.pdf

http://www.roboticseducation.org/about/

http://www.roboticseducation.org/about/

www.asee.org/search/proceedings?search=session_title%3A%22Engineering+in+Middle+Schools%22+AND+conference%3A%222007+Annual+Conference+%26+Exposition%22




Silk, EM (2011). Resources for learning robots: Environments and framingsconnecting math in robotics (Doctoral dissertation). Retrieved from ACMDigital Library.(AAI3485771)

Silk, EM, Higashi, R, Shoop, R, & Schunn, CD. (2010). Designing technologyactivities that teach mathematics. The Technology Teacher, 69(4), 21–27.

Silk, EM, Schunn, CD (2008). Core concepts in engineering as a basis forunderstanding and improving K-12 engineering education in the UnitedStates. Report submitted to the National Academy of Engineering Committeeon Understanding and Improving K-12 Engineering Education in the UnitedStates. Retrieved from http://elisilk.net/research/SilkSchunn2008a-NAE-Final-Draft.pdf

Vollstedt, AM, Robinson, M, & Wang, E. (2007). Using robotics to enhance science,technology, engineering, and mathematics curricula. In Proceedings ofAmerican Society for Engineering Education Pacific Southwest annualconference, Honolulu: Hawaii

Weaver, R, Junker, B. W (2004). Model specification for cognitive assessment ofproportional reasoning. Department of Statistics, Paper 171. Retrieved fromhttp://repository.cmu.edu/statistics/171

Zimmerman, BJ, & Martinez-Pons, M. (1990). Student differences in self-regulatedlearning: relating grade, sex, and giftedness to self-efficacy and strategy use.Journal of Educational Psychology, 82(1), 51–59. doi:10.1037/0022-0663.82.1.51.

Submit your manuscript to a journal and benefi t from:

7 Convenient online submission

7 Rigorous peer review

7 Immediate publication on acceptance

7 Open access: articles freely available online

7 High visibility within the fi eld

7 Retaining the copyright to your article

Submit your next manuscript at 7 springeropen.com

http://elisilk.net/research/SilkSchunn2008a-NAE-FinalDraft.pdf

http://elisilk.net/research/SilkSchunn2008a-NAE-FinalDraft.pdf

http://repository.cmu.edu/statistics/171

Documents

Case studies of a robot-based game to shape interests and ... · non-traditional methods or multi-modal methods of in-struction can be more effective (Kroesbergen and Van Luit 2003;