Teaching and LearningComputer Science SortingAlgorithms With MobileDevices: A Case StudyIVICA BOTICKI,1 ANTE BARISIC,1 SERGIO MARTIN,2 NEVEN DRLJEVIC1

1Faculty of Electrical Engineering and Computing, Department of Applied Computing, University of Zagreb, Zagreb,

Croatia

2Electrical and Computer Engineering Department, UNED (Spanish University for Distance Education), Madrid, Spain

Received 3 December 2011; accepted 20 February 2012

ABSTRACT: In this article, we present a semester-long study on using smartphone devices in computing

engineering education. We developed Sortko, an Android-based smartphone application for learning sorting

algorithms, an important undergraduate computer science topic. The application consists of four main compo-

nents—the module for interactive sorting, the scaffolding module, the motivational module, and the graphical

user interface module, each with a distinct role of helping students in learning sorting algorithms. Our re-

search methodology included data collection with administered two surveys, collected exam results and

recorded application usage data. Analysis of the collected data shows our approach is an effective way of

learning sorting algorithms. � 2012 Wiley Periodicals, Inc. Comput Appl Eng Educ; View this article online at

wileyonlinelibrary.com/journal/cae; DOI 10.1002/cae.21561

Keywords: mobile learning; engineering education; sorting algorithms

INTRODUCTION

This study examines educational benefits of smartphone devices

in higher education. In our approach students use Android [1]

devices as a support in the process of learning sorting algo-

rithms. Sorting, or putting series of elements in appropriate or-

der, is an important computer science topic and presents an old

and well-defined computer science problem. Mastering sorting

algorithms is especially important for future software engineers

regardless of the specific domain they work in.

The growing popularity of smartphone devices can be

clearly observed in everyday student life. Although they are

mostly used for fun and informal ways of learning (e.g., access

to Wikipedia, YouTube, etc.), scientific studies have shown

the educational value of these devices. Based on an informal

observation of our students’ interest in learning with smart-

phone devices and by taking into account behaviorism and so-

cial—constructivist theory of learning, we are suggesting a

design of a smartphone application for learning sorting algo-

rithms which, amongst others, includes a scaffolding module

designed to assist students while performing sorting and a moti-

vational module to additionally motivate students in using the

application.

Normally, students learn sorting by using a paper and pen-

cil while simultaneously watching sorting algorithm program

code. This tiring and ineffective procedure often results in dif-

ferent kinds of errors such as students learning their own ver-

sion of a sorting algorithm which does not match the real one

(the sequence of steps while sorting elements is not adequate).

We believe that technology can restructure this troublesome and

ineffective process of learning thereby improving students’

knowledge of sorting algorithms. To verify our assumptions we

are exploring these research questions:

� The students who used the application achieve better

academic results than the students who did not use the

application.� The scaffolding module is able to assist students in the

process of sorting.� The motivational module motivates students on further

application use.

These research questions will be explored by analyzing

results of standard exams, usage logs, and specially designed

Correspondence to I. Boticki (ivica.boticki@fer.hr).

� 2012 Wiley Periodicals, Inc.

1

questionnaires querying our students’ perception on usage and

usefulness of the Sortko application and the approach in

general.

Learning and Playing With Mobile Devices

An early research conducted in England as early as in 2005 had

shown that more than 95% of students possess at least one mo-

bile device [2], which opens up a space for their use in the

process of educating young people. As for the effectiveness of

the approach, another study presents a learning trail in which

students were equipped with 100 Palm Pads stating that 90% of

teachers had a positive attitude regarding the impact of this

type of technology on students and on their apprehension of

learning content [3,4].

Apart from mobile learning [5,6], game-based learning

examines the process of studying which is catalyzed and stimu-

lated with the elements that are considered to be the standard

parts of games (i.e., awards, collecting points, reaching higher

levels, and achieving high scores in computer games). These

motivate players to be involved in the game more thus enhanc-

ing and accelerating their learning process through repeated

game play [7].

The crucial part of such approaches is the united goal of

the game and of the learning process [8,9]. Once that is

achieved, learning becomes entertainment and students get mo-

tivated to learn at home and to take true interest in the course

contents making the educational process more efficient and

self-regulated [10]. What is more, by employing new pedago-

gies which integrate learning with game play, teaching becomes

more fun as well.

A Paper and Pencil Approach to Learning SortingAlgorithms

The topic of learning sorting algorithms is a part of the Algo-

rithms and Data Structures course at the Faculty of Electrical

Engineering and Computing, University of Zagreb, Croatia and

has so far been taught in a paper-and-pencil fashion. Typically,

various types of sorting algorithms (such as the Bubble sort

algorithm which ‘‘bubbles out’’ the greatest number in each

algorithm pass) [11,12] are visualized by drawing a series of

algorithm passes, where each pass is drawn as an array of num-

bers on the blackboard or a sheet of paper. After each algorithm

pass, a new sequence of numbers is written down again

(Fig. 1). As a consequence, during exams, students’ knowledge

of sorting algorithms is tested in the same way.

This repetitive and cumbersome way of learning often

results in errors (i.e., a wrong sequence of algorithm steps is

learned) and distracts students from actually understanding the

algorithm big picture. Another problem that arises from this

type of learning is that there is no easy way a student can check

the accuracy of her proposed sorting algorithm passes. What is

more, students sometimes get a correct (sorted) sequence even

when a wrong set of algorithm steps is applied. For example,

while applying Quick sort algorithm and identifying the pivotal

element, a student can decide not to follow the step-to-step ap-

proach of the algorithm and just position all smaller numbers

on the left-hand side of the pivotal element and all greater num-

bers on the right-hand side of the pivotal element. Although

conceptually correct and with a correct final solution, this way

of sorting does not indicate whether the student actually knows

the precise sequence of steps of the Quick sort algorithm and

hence is graded with lower marks.

The Use of Animations in Learning Sorting Algorithms

The use of animation materials as learning resources has been

advocated by many researchers [13] and there are a number of

studies dealing with overcoming the typical difficulties in learn-

ing sorting algorithms with the use of animations [14,15]. Thus,

many authors [16–19] have provided animated resources to

their students in order to enhance the concepts acquisition.

Sutinen [20] introduced a Web-based tool, Jeliot, providing

simple animations helping students in learning and understand-

ing algorithms and data structures. Kordaki [21] performed a

pilot where secondary school students used a web-based appli-

cation helping them in understanding different sorting algo-

rithms during in-class learning.

An algorithm animation helps students in the visualization

of a specific sorting algorithm but does not allow them to try

out sorting on their own [22]. For example, www.sorting-algo-

rithms.com web page [23] with a parallel animation of several

sorting algorithms on the same unsorted number sequence.

With this simulation several sorting algorithm behaviors can be

observed on a randomly generated, almost sorted, reversed or a

sequence with repeated numbers. Figure 2 shows the Sort Ani-

mation web page which, in addition to the sorting algorithm

operation animation, allows a parallel view of the programming

code being executed during a specific animation step. The

application also shows temporary algorithm variables and pro-

vides users with a possibility of adjusting the sorting sequence

length.

In general, animations help viewers track patterns and ob-

serve relationships, which is claimed to improve learning [24].

However, some researchers challenge this stating that the use of

PC-based animations for algorithm visualization does not nec-

essarily lead to better concepts acquisition than with static dia-

grams, especially in the case of students needing to make

Figure 1 Traditional visual representation of the Quick sort sorting

algorithm operation while sorting an array of numbers. [Color

figure can be viewed in the online issue, which is available at

wileyonlinelibrary.com.]

2 BOTICKI ET AL.

predictions of the coming animation sequence [13,15,25,26]

and even suggest that animations distract students. Building on

the rich amount of knowledge in learning with the help of ani-

mations we propose a design in which instructional agency

shifts toward the students’ side. By allowing students to explore

the rich space of both incorrect and correct solutions while sort-

ing, we aim to utilize technology capabilities to provide them

with scaffolds in an experience-driven learning approach.

MATERIALS AND METHODS

Sortko System in Use

The smartphone mobile learning application Sortko, designed

as a part of this study, provides students with a graphical user

interface with an option of choosing a sorting algorithm (e.g.,

bubble sort, shell sort, quick sort, insertion sort, etc.) and gener-

ates a random sequence of numbers to be sorted by applying a

sequence of specific interaction gestures. During the sorting

process, the system informs students about their success in per-

forming so-called ‘‘switches’’ (element exchanges) and pro-

vides them with textual help messages facilitating further

progress. For each game a student gets rewarded with points

that are then summarized on a joint public list.

The graphical user interface consists of three main views:

initial view, sorting view, and the result view, all made in hori-

zontal mode to maximize display space and usage ergonomics

(left-hand side of Fig. 3a).

The sorting view contains a sequence of randomly gener-

ated numbers (Fig. 3a, callout 1). While students sort by apply-

ing gestures (Table 1), the system displays contextual textual

help messages (Fig. 3a, callout 3). After each step (pass) of a

sorting process, the textual help message changes therefore

directing students toward the next required action. The current-

ly used algorithm is shown to users as well (Fig. 3a, callout 4)

indirectly determining the number of points a student will be

assigned according to the predefined formula (details of the mo-

tivational module are described in the Motivational Module

Section).

The Sortko application contains an auxiliary visual ele-

ment positioned separately from the sorting array (Fig. 3a,

callout 2). It is used to hold a variable with a temporarily

element value stored until certain algorithm condition gets

fulfilled.

After sorting has completed, the results are sent to the

server side and the complete ranking list is shown to the sorter

(Fig. 3b).

Sortko Application Model

Sortko System Architecture. Sortko application is built for An-

droid operating system which includes a set of programming

libraries corresponding to the elementary Java programming

language library [1]. Every Android application runs in its own

process, with its own instance of the Dalvik virtual machine

(VM). Dalvik [27] is a purpose-built VM designed specifically

for Android taking into account inherent mobile device limita-

tions such as short battery life and processing power. Sortko

application is created using Eclipse integrated development

tools (IDE) equipped with the Android Development Tools

plugin (ADT plugin) which allows for an easy setup of Android

projects, application UIs and for the Android Framework API

component reuse [28].

The Sortko system is based on the client–server architec-

ture model and is divided into the following components: data

access layer, the domain model, web component, service com-

ponent, and the smartphone client application (Fig. 4). The

server side of the system consists of a database, data access

layer, web services, and a web site. The web part resides on the

data access layer and can be accessed via desktop computers

and mobile devices.

The client part of the system is developed as an Android

smartphone application and consists of the following modules

(Fig. 5):

� The Interactive sorting module;� The Scaffolding module providing help messages;

Figure 2 Sort Animation web page with a parallel view of a sorting

animation and the appropriate algorithm execution (http://www.cs.oswego.

edu/�mohammad/classes/csc241/samples/sort/Sort2-E.html). [Color figure

can be viewed in the online issue, which is available at wileyonlinelibrary.

com.]

Figure 3 Graphical user interface of the Sortko Android smartphone application: (a) sorting view and (b) results

view. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

SORTING WITH MOBILE DEVICES 3

� The Motivational module for awarding students;� The User interface module.

Graphical User Interface Module. The Interactive sorting

module verifies whether sorting steps provided by a student are

valid for a given sorting algorithm. The Scaffolding module

helps students in solving typical dilemmas in the sorting pro-

cess and displays help messages after an error in sorting has

been identified. It directs users toward choosing the right sort-

ing solution by illustrating the logic behind a sorting problem,

which is crucial in understanding sorting algorithms. The Moti-

vational module tracks the successfulness of sorting according

to different parameters (speed, accuracy, etc.) and awards points

to users accordingly. The summary points and ranks are pre-

sented in form of a web page accessible to all students.

The client side application architecture follows the Model-

View-ViewModel pattern [29]. It consists of a view, which col-

lects user input and forwards it to the viewmodel and which

actively pulls the data from the viewmodel by using the data

binding technique. This pattern facilitates desirable software de-

velopment techniques such as the separation of concerns

(decoupling), unit testing, and test driven development. View-

model encompasses all the benefits of the patterns such as Mod-

el-View-Controller and in the same time has an additional

benefit: the simplification coming from the declarative binding

transferring data bidirectionally between viewmodel and the

view.

Interactive Sorting Module. The Interactive sorting module

contains specialized implementations of sorting algorithms used

to verify whether a user has chosen the accurate algorithm

steps. The base class inherited by all specialized algorithm

implementations is the abstract Algorithm.java class. Concrete

implementations of algorithms are (1) BubbleSort.java, (2)

InsertionSort.java, (3) QuickSort.java, (4) SelectionSort.java,

and (5) ShellSort.java (Fig. 6).

Scaffolding Module. The Scaffolding module helps students in

sorting by giving them contextual textual help messages with

the instructions on how to continue or on how to correct an

error. It is linked with the Interactive sorting module and pre-

serves the current state of the sorting algorithm currently in use

(Fig. 7).

The contextual text help messages are implemented

according to each algorithm’s specifics and capture current al-

gorithm position. For example, textual help messages for the

Bubble sort algorithm are provided by the BubbleSortPosition.-

java class.

Motivational Module. The Motivational module assigns points

to students which are then sent to the server-side and there get

summarized. The amount of points a student is awarded

depends on the accuracy and the speed of her actions and the

type of algorithm used. At a microlevel (one user action or one

element exchange), points for a successful user action are

awarded according to the following formula:

User Action Points ¼ 1; 000þ Alg Weight� 10; 000

� 1

½2� ðAlg Weight=10Þ�Switch Duration

The formula is designed specifically for this project in or-

der to reward the use of more difficult algorithms. It consists of

an exponential decreasing function starting with the maximum

number of points, which decreases over time. For more

Table 1 Typical Sorting Gestures

Sorting gesture Sorting gesture description

Sorting gesture

algorithm coverage Note

Element switch Two sequence elements switch their positions Bubble sort, Selection sort Action sequence not important

Element switch

(with itself)

Element gets switched with itself, no visible

sequence change afterwards

Quick sort, Selection sort

Element copy One element value is copied into another element Shell sort, Insertion sort Action sequence is important

Figure 4 Sortko system architecture. [Color figure can be viewed in

the online issue, which is available at wileyonlinelibrary.com.] Figure 5 Client side of the Sortko system.

4 BOTICKI ET AL.

complex algorithms, the function is declining more slowly but

starts from a much larger number of points, therefore leaving

more time for choosing an algorithm step to students. In case

of simple sorting algorithms, the function is adjusted so that it

rewards speed and accuracy, and has a smaller contribution to

the overall amount of collected points. Such a reward system

should direct users toward using more complex algorithms.

The duration of one element switch (Switch_Duration) is

the time from the last successful element switch till the current

element switch. The formula constant 1,000 represents minimal

points a student can get per one action (one element exchange),

while the constant 10,000 represents the maximum number of

points a user can get per one action. Constants 2 and 10 are

used to model exponential function used to reduce the number

of points awarded per user action as the exchange duration

increases. As a consequence, a student gets a minimum of

1,000 points for a successful action no matter how long it took

to complete, and a maximum of Alg_Weight � 10,000 points

for a successful action with the shortest possible element ex-

change duration (if her action happened really fast). The algo-

rithm weight (Alg_Weight) is a constant specific to each sorting

algorithm used to reward students with more points when sort-

ing with Quick sort and Shell sort algorithms. For an unsuc-

cessful action users get penalized with a fixed amount of 1,000

points. The final game points are calculated by summing up all

points for successful actions and by deducting all points coming

from the unsuccessful actions, therefore forming total single

game points.

Figure 8 shows two graphs with the visualization of the

User_Action_Points formula for the two distinct algorithm

weight (Alg_Weight) constants 1 (Bubble Sort, Insertion Sort

and Selection Sort) and 5 (Quick Sort and Shell Sort algorithms).

The graphs show that the function gives more points to the

more demanding algorithms. For example, for a switch interval

of 5 s, a student is awarded with around 1,300 points in the first

case (Alg_Weight ¼ 1) and with around 7,500 points in the

second case (Alg_Weight ¼ 5).

RESEARCH DESIGN, DATA COLLECTION,AND RESULTS

Sortko was used in the summer semester of the academic year

2010/2011 of the ‘‘Algorithms and Data Structures’’ course

taught as a part of the undergraduate study at the Faculty of

Electrical Engineering, University of Zagreb. Prior to the

students being formally introduced with the topic of learning

sorting algorithms, the Sortko system was presented and the

initial survey was administered.

During the classes covering sorting algorithms students

were able to download and install the Sortko application onto

their Android smartphones from the Sortko website [30].

They were enticed to do so by the teacher, but there was no

curriculum integration of the Sortko activities, neither were

there additional merits for students. Sortko was deployed as an

‘‘add-on’’ and its value is examined from that perspective

throughout this section of the paper. Although, initially, the

application was available for download as an Android-specific

‘‘apk’’ file, the new Sortko version can be found on the Android

Market and downloaded as any other Android smartphone

application [31].

After the topic of sorting algorithm was completed, the

students took their midterm exams. In addition to the midterm

exam results, the log information on students’ activity in the

Sortko system was collected and analyzed. As the final data

collection mechanism in this study, another survey was admin-

istered in order to gather the overall students’ experiences.

The Initial Survey

Prior to the Sortko application release, the initial anonymous

survey was administered. One hundred twenty-two out of 596

students enrolled in the course took part in this survey. It con-

sisted of 20 questions mainly targeting students’ affinities to-

ward the use of technology in learning, availability of Android-

based smartphones, etc. (Tables 2 and 3).

Figure 6 Class diagram showing the main classes of the Interactive sorting module. [Color figure can be viewed in

the online issue, which is available at wileyonlinelibrary.com.]

Figure 7 Class diagram showing the main classes of the Scaffolding module. [Color figure can be viewed in the

online issue, which is available at wileyonlinelibrary.com.]

SORTING WITH MOBILE DEVICES 5

The survey results show that sorting algorithms present a

new topic to the students. Although the majority has heard

about sorting, as much as 69% claim they do not possess opera-

tional knowledge of sorting. This finding eliminates the tertium

quid effect of the prior knowledge of sorting that might inter-

fere with our study results. Additionally, we were interested in

finding out whether our students were keen on accepting the

new way of studying with smartphones. As the survey shows,

they feel they learn best when they do it home alone studying

course literature (67%) with only 28% applying some form of

collaborative learning. This does not come as a surprise consid-

ering half of the students are not satisfied with the formal lec-

tures as they are currently done by the teachers in classrooms.

This finding not only opens up a new space for curriculum in-

novation, but also cautions us as our students are generally not

satisfied with the current learning and teaching system.

As a digital teaching tool, Sortko differs from the tools

students are used to in their prior education. This does not pres-

ent a problem to our students who believe that the technology

can be used for learning (93%) and that it can enhance educa-

tion (96%). They also believe sorting algorithms can be learned

by using computers (90%) and as much as 85% believe that it

can be done better than without computers. As far as smart-

phones are concerned, the students are unsure whether they can

help in learning. However, the students who already own a

smartphone gave more optimistic responses (AVG ¼ 3.41)

compared to the students who do not have a smartphone device

(AVG ¼ 2.86), with the mean difference of 0.55 being

significant t(120) ¼ �2.08, P < .05 two-tailed, r ¼ .18, repre-

senting a small-sized effect. Similar results were acquired for

the general questions on use of technology in education.

In total 39% of the surveyed students have their own

smartphone device and 62% of smartphone users own an

Android-based device, with version 2.2 being the most frequent

(52%).

Sortko Usage and Midterm Results Analysis

After each successful sequence sorting, Sortko sends student

identification, sorting algorithms applied, and the score awarded

to the student to the server-side storage. That data is used to

analyze what kind of sorting algorithms are most frequent and

whether usage correlates with the standard exam results. The

log data shows the most frequently used sorting algorithm was

Bubble sort (31% of the total sequence sorting done), followed

by Selection Sort (27%), Quick Sort (17%), Insertion Sort

(15%), and the least used Shell sort (10%). We explain this

difference with the two main reasons: Bubble sort is the sim-

plest and usually the first algorithm taught.

The course ‘‘Algorithms and data structures’’ includes two

midterm exams and a final exam. Sorting algorithms are taught

as the middle part of the course and are assessed with an as-

signment in the second midterm exam, such as this:

A sequence of numbers 8, 6, 4, 9, 2, 7, 5, 0, 3, 1 needs

to be sorted using the Quick sort algorithm. Illustrate the

sorting algorithm behavior by producing a new sequence

after each element exchange and specially mark the

numbers relevant for the next step (exchanges, pivotal

elements, median approximation). The pivotal element

needs to be chosen with the method of median approxi-

mation of the first, middle, and last sequence element.

Figure 8 User_Action_Points formula graphs for the two distinct

Alg_Weight constants.

Table 2 Initial Survey Items and Summarized Students’ Responses—Likert Scale Question Items (N ¼ 122)

Survey item AVG SD

Before the sorting algorithms lectures started I already knew sorting algorithms well 1.98 1.083

During the ASP course lectures I am able to learn course contents well 3.17 1.197

I learn the most by studying literature (e.g., slides, books, textbooks, exams) at home 3.88 1.182

I learn the most when studying with colleagues in a group 2.66 1.270

I like the idea of using technology in learning programming 4.28 0.884

I believe the technology can help in improving education 4.62 0.647

I believe the technology can be used for learning 4.56 0.793

I believe there are advantages to using smartphones in education compared to classic desktop computers

due to their characteristics (easy to carry, instant connectiveness, etc.)

2.99 1.269

I believe sorting can be learned with the help of computers 4.41 0.747

I believe sorting can be better learned with the use of computer than without the use of computers 4.29 0.922

I would like to participate in the research on learning sorting algorithms using smartphones 3.09 1.477

Table 3 Initial Survey Items and Summarized Students’

Responses—Yes/No Question Items (N ¼ 122)

Survey item Yes No

I own a mobile phone (of any kind—cell

phone, smartphone, iPad, etc.)

120 (98%) 2 (2%)

I own a smartphone (Android, iPhone,

Windows Phone 7, Blackberry, etc.)

47 (39%) 75 (61%)

I own a smartphone with an Android

operating system

29 (24%) 93 (76%)

6 BOTICKI ET AL.

In our study we were interested in finding out the correla-

tion between the three main data sources: the total points stu-

dents had on the Sortko system after the semester was done, the

total points students had on the Quick sort algorithm on the

Sortko system after the semester was done, and the results from

the above presented assignment (standard midterm exam task

4). The correlations are given in Table 4.

Since the standard exam assignment included Quick sort

algorithm, we calculated the correlation between the application

usage frequency and the standard exams assignment results and

got a fairly high value of 0.47 with a significance of P < 0.05.

Furthermore, the correlation between the Sortko total rank and

the Quick sort assignment results gives a similarly high value

of 0.53. Therefore, we became confident that Sortko usage

means better standard exam results and explored this hypothesis

by applying the independent sample t-test to compare between

groups according to the mean values of usage frequency, total

Sortko points, and the Quick sort Sortko points (Table 5).

The analysis shows there is a significant difference be-

tween exam results when comparing Sortko users and Sortko

non-users of around 30% according to all three criteria. The

effect was large-sized of r ¼ 0.52, 0.70, and 0.65 for total us-

age frequency, total Sortko points, and Quick sort Sortko points

respectively. As a control mechanism, there are no significant

differences between the Sortko users and Sortko non-users

when comparing their second midterm and first and second

midterm exam results. This proves our initial hypothesis that

Sortko users achieve better standard academic results than

Sortko non-users.

The Final Survey

The final survey was done after the lectures and the second

midterm exam were completed. It was conducted on a volun-

tary basis attracting 29 students who used the Sortko system.

They were asked how practical the application is when learning

sorting algorithms, how well individual application modules

were structured, etc. (Table 6). Additionally, as a part of the

survey, we were interested in finding out whether and how often

our students used the application. Sixteen out of 29 students

claim they used Sortko once, while 27 out of 29 responded they

used it more than once during the semester.

In the analysis of the final survey, we separated the stu-

dents into two groups according to the academic success and

the usage frequency criteria. We wanted to identify survey

items for which these groups of students gave significantly dif-

ferent responses.

Generally, the students feel Sortko helped them in learning

sorting algorithms and in achieving better results in standard

exams which is also confirmed by the quantitative analysis pre-

sented in the previous section. They feel the Sortko Scaffolding

module helped them in dealing with doubts while sorting, while

the motivational module attracted a diverse set of responses

which is analyzed in detail in the following sub-section.

In order to further elaborate on the students’ answers to

the survey questions, they were separated into two groups

according to two main criteria: Their academic success and

their Sortko usage frequency. By doing this we were able to

explore whether the difference in students’ success and the

number of times they used Sortko affects their perception as

expressed through the administered survey. When comparing

students according to their academic ability we found out there

is a significant difference between more and less academically

successful students on only one survey item:

� ‘‘The Sortko help message system helped me to learn

sorts better’’ (P < 0.05, one-tailed significance level,

r ¼ .19, small-sized effect).

When comparing students’ answers to the survey questions

according to the Sortko usage frequency, we found out there is

a significant difference between students who used the applica-

tion a lot and the ones who used it less on the following survey

items:

� ‘‘Sortko application helped me in learning sorts’’

(P < 0.05, two-tailed significance level, r ¼ .40, large-

sized effect).� ‘‘Sortko application helped me in achieving better results

on exam/quizzes/short lecture assessments’’ (P < 0.05,

two-tailed significance level, r ¼ .33, medium-sized effect).� ‘‘The Sortko rank list motivated me to compete and use

the Sortko application multiple times’’ (P < 0.05, two-

tailed significance level, r ¼ .70, large-sized effect).

Motivational Module Effectiveness Analysis

Sortko’s motivational module was used to introduce a game ele-

ment into the sorting activities. The rewards system was set up

so that it stimulates using more complex algorithms such as the

Quick sort or Shell sort by simply awarding such sorts with

more points compared to the simpler algorithms.

Table 4 Correlations Between Sortko Points, Sortko Usage, and

Midterm Exam Results

Total Sortko

points

Quick sort

Sortko points

Midterm 4th

exam points

Quick sort sorting frequency 0.751�� 0.802�� 0.473�Total Sortko points — 0.788�� 0.528�Quick sort Sortko points — — 0.499�

�P < 0.05 (two-tailed).��P < 0.01 (two-tailed).

Table 5 Independent Sample t-Test Exploring the Difference in Academic Results Between Sortko Users and Sortko Non-Users (N ¼ 24)

Total usage frequency Total Sortko points Quick sort Sortko points

Fourth standard exam results (mean difference

between groups in % of the assignment total)

29,076� 36,333� 31,143�

Second midterm standard exam 4,711 3,833 3,657

First and second midterm exam results 3,956 3,167 3,171

�P < 0.05 (two-tailed).

SORTING WITH MOBILE DEVICES 7

The correlation of students’ responses on the survey ques-

tion ‘‘Rank list motivated me to compete and use Sortko multi-

ple times’’ (AVG ¼ 4.29) and the usage frequency for the

students who actually used the application is r ¼ .35, present-

ing a medium-sized effect. Sortko power users gave more posi-

tive answers to this question which confirms the rewards

system motivated students to compete and use Sortko multiple

times.

An interesting issue arises in the final survey analysis

which shows that the application use frequency negatively cor-

relates with the perception of adequacy of the rewards system

(r ¼ �.37). However, by taking into account the final survey

results analysis on the question item ‘‘The Sortko rank list mo-

tivated me to compete and use the Sortko application multiple

times’’, we were able to identify there is a significant difference

of 1,894 between frequent and non-frequent Sortko users

(P < 0.001, two-tailed significance level). The explanation for

such responses was looked for through informal personal inter-

views in which we found out that in some cases points get

awarded depending on luck. The main reason for that is the

initial sorting sequence generation algorithm which in some

cases generates sequences which are easy to sort. This leads to

an interesting situation where a couple of Sortko power users

had an extremely negative opinion regarding the rewards sys-

tem. Regardless of the perceived unfairness, the correlation co-

efficient between the usage frequency and the perceived

motivational effects of the rewards system is relatively high

(r ¼ 0.33).

A surprising effect of the motivational module emerged

after the final survey was administered and the study was for-

mally completed with additional 519 sequences sorted (27% of

the totally sorted sequences). In addition to the pre-exam time,

there is a usage peak between May 12, 2011 and May 22, 2011,

a period after the final exam (Fig. 9). This sustained usage tells

us the standard exam is not the only motivational element for

using Sortko and confirms the benefits of the Motivational

module.

CONCLUSIONS

This article described a case study exploring the benefits of

using mobile devices in supporting teaching and learning of

sorting algorithms, an important computer science topic. We

proposed an approach in which an Android mobile application

was designed and made available to students to be used in and

out of classrooms. As a contrast to the conventional mode of

teaching and learning, this application allowed students to en-

gage in learning sorting algorithms in a location-independent

way—both in and out of classrooms. By leveraging the existing

body of knowledge dealing with the use of animation in teach-

ing and learning, we designed an interactive application which

scaffolds students in trying out sorting on their own. Our design

Table 6 Final Survey Items and Summarized Students’ Responses—Likert Scale Question Items (N ¼ 29)

Survey item AVG SD

Sig.

(two-tailed)

ac. succ.

Mean

diff.—ac.

succ.

Sig.

(two-tailed)—

usage freq.

Mean

diff.—usage

freq.

Before the lectures on sorting in ASP course started I knew sorts well 1.86 0.953 0.128 0.543 0.850 �0.071

Sortko application helped me in learning sorts 4.28 0.922 0.735 �0.119 0.037�� 0.727

Sortko application helped me in achieving better results on

exam/quizzes/short lecture assessments

3.86 1.246 0.547 �0.286 0.002�� 1.247

By using Sortko application throughout a longer period of time,

the sorting course contents are better mastered

4.21 1.146 0.548 �0.262 0.220 0.545

The Sortko help message system helped me to learn sorts better 4.31 0.761 0.098� �0.462 0.435 0.232

The Sortko help message system helped me in clearing some

doubts regarding sorts

4.00 0.845 0.192 �0.414 0.179 0.439

Sortko help messages clearly direct on the application usage and

identify errors during sorting

3.72 1.066 0.466 �0.295 0.490 �0.288

The Sortko rank list motivated me to compete and use the

Sortko application multiple times

3.55 1.639 0.778 0.176 0.000��� 1.894

The Sortko reward (points) system is adequate and fair 3.48 1.379 0.950 0.033 0.428 �0.485

The Sortko reward (points) system motivates me to

additionally use the application

3.72 1.306 0.382 �0.433 0.384 0.444

I like Sortko’s graphical design 4.45 0.827 0.751 0.100 0.159 0.449

The Sortko application is easy and intuitive to use 4.34 0.936 0.947 0.024 0.195 0.470

Now many points did you collect at the ASP course so far? 33.81 10.47

�P < 0.05 (one-tailed).��P < 0.05 (two-tailed).���P < 0.001 (two-tailed).

Figure 9 Sortko usage peaks (summer 2011 semester).

8 BOTICKI ET AL.

acknowledged the role of motivation in learning by awarding

student with points which were their accumulated on the com-

mon rank list stimulating prolonged application use.

We found out that student like using technology for learn-

ing, although they are not all sure that smartphones specifically

can help in learning. However, we were able to find out that the

students who already own a smartphone recognize its educa-

tional benefits better than the non-owners. As far as the Sortko’s

motivational module is concerned, we are able to report that

our students found it effective and that it enticed students on a

prolonged system use. On the other hand, standard exam results

analysis shows Sortko users achieve better results compared to

Sortko non-users.

We plan to upgrade Sortko for the upcoming academic

year, focusing on improving the Motivational module by reduc-

ing the luck factor in the initial number sequence generation as

well as creating a skill-based ranking mechanism in addition to

the purely cumulative one. This should make the Motivational

module more effective in motivating students on using Sortko

and neutralize the annoyance exhibited by power users who fa-

vor skill-based competition, as shown by the analysis. Due to

the rapid growth of the smartphone market porting Sortko onto

other prevailing mobile platforms, such as iPhone or Windows

Phone 7, should allow us to reach wider audiences and affirm

Sortko as an educational training tool for mastering sorting

algorithms.

Although the application and the results were encouraging,

we cannot emphasize the role of curriculum integration more.

Since the Sortko application and this study were done as an

add-on onto the curriculum, we were not able to motivate the

majority of students to participate. It is our belief that similar

educational interventions have to be employed as well integrat-

ed curriculum elements and not as add-ons to achieve the full

and sustained educational power.

To sum up, mobile learning applications such as Sortko

are able to engage students and can serve as a motivational

vehicle in computer science engineering motivation. Students

are open to such interventions and educators should leverage on

the intrinsic benefits of mobile technologies in developing their

teaching and assessment strategies.

REFERENCES

[1] Android-Online, Android developers guide, Retrieved February

15, 2012, From http://www.android.com/developers/.

[2] L. Naismith, P. Lonsdale, G. Vavoula, and M. Sharples, Literature

review in mobile technologies and learning, report no. 11, Future-

lab, Bristol, United Kingdom, 2005.

[3] H. B. Andreu and M. Nussbaum, An experimental study of the

inclusion of technology in higher education, Comput Appl Eng

Educ 17 (2009), 100–107.

[4] J. Roschelle, Keynote paper: Unlocking the learning value of wire-

less mobile devices, J Comput Assist Learn 19 (2003), 269–272.

[5] C. Romero, S. Ventura, and P. de Bra, Using mobile and web-

based computerized tests to evaluate university students, Comput

Appl Eng Educ 17 (2009), 435–447.

[6] G. Vavoula and M. Sharples, Challenges in evaluating mobile

learning, Proceedings of the mLearn 2008 Conference, Shropshire,

United Kingdom, 7th October 2008.

[7] M. Ebner and A. Holzinger, Successful implementation of user-

centered game based learning in higher education: An example

from civil engineering, J Comput Educ 49 (2007), 873–890.

[8] A. A. Deshpande and S. H. Huang, Simulation games in engineer-

ing education: A state-of-the-art review, Comput Appl Eng Educ

19 (2011), 399–410.

[9] M. Milrad and M. Jackskon, Designing and implementing educa-

tional mobile services in university classrooms using smart phones

and cellular networks, Int J Eng Educ Mobile Technol Eng Educ

23 (2007), 84–91.

[10] T. A. Philpot, R. H. Hall, N. Hubing, and R. E. Flori, Using games

to teach statics calculation procedures: Application and assess-

ment, Comput Appl Eng Educ 13 (2005), 222–232.

[11] A. Drozdek, Data structures and algorithms in Cþþ, Course tech-

nology, Boston, 2005.

[12] M. A. Weiss, Data structures and algorithm analysis in C, Addison

Wesley, Reading, MA, 1997.

[13] M. D. Byrnea, R. Catramboneb, and J. T. Stasko, Evaluating ani-

mations as student aids in learning computer algorithms, Comput

Educ 33 (1999), 253–278.

[14] S. R. Hansen, N. H. Narayanan, and D. Schrimpsher, Helping

learners visualize and comprehend algorithms, Interact Multimedia

Electron J Comput Enhanced Learn 1 (2000).

[15] C. Kann and C. Hundhausen, Integrating algorithm visualization

technology into an undergraduate algorithms course: ethnographic

studies of a social constructivist approach, Comput Educ 39

(2002), 237–260.

[16] R. M. Baecker, Sorting out sorting. Film. University of Toronto,

M. Kaufmann Publishers, San Francisco, USA, 1981.

[17] B. Deliktas, Computer technology for enhancing teaching and

learning modules of engineering mechanics, Comput Appl Eng

Educ 19 (2011), 421–432.

[18] H. L. Dershem and P. Brummund, Tools for web-based sorting

animation, ACM Special Interest Group Comput Sci Educ Bull

Inroads 30 (1998), 222–226.

[19] T. L. Naps, Algorithm visualization in Computer Science Labora-

tories, Proceedings of the 21st SIGSE Technical Symposium on

Computer Science Education Special Interest Group for Computer

Science Education Bulletin Inroads, Washington, USA, February

1990.

[20] E. Sutinen, J. Tarhio, and T. Tera, Easy algorithm animation on

the Web, Multimedia Tools Appl 19 (2003), 179–194.

[21] M. Kordaki, M. Miatidis, and G. Kapsampelis, A computer envi-

ronment for beginners’ learning of sorting algorithms: Design and

pilot evaluation, Comput Educ 51 (2008), 708–723.

[22] S. Price and Y. Rogers, Let’s get physical: the learning benefits of

interacting in digitally augmented physical spaces, J Comput Educ

15 (2004), 169–185.

[23] D. R. Martin, Sorting algorithm animations, 2007, From http://

www.sorting-algorithms.com/.

[24] G. G. Robertson, S. K. Card, and J. D. Mackinlay, Information

visualization using 3D interactive animation, Commun ACM 36

(1993), 57–71.

[25] J. S. Gurka, Pedagogic aspects of algorithm animation, Unpub-

lished PhD Dissertation, University of Colorado, Colorado,

1996.

[26] J. Stasko and E. Kraemer, A methodology for building applica-

tion-specific visualizations of parallel programs, J Parallel Distrib

Comput 18 (1993), 258–264.

[27] D. Bornstein, Presentation of Dalvik VM Internals, Google,

2008.

[28] Eclipse-Online, About the Eclipse Foundation, Retrieved February

15, 2012, From http://www.eclipse.org/org/.

[29] S. F. Allam, SOA Design pattern for distributing the object model

in n-tier applications, Innovations and advanced techniques in sys-

tems, computing sciences and software engineering, Springer,

Netherlands, 2008, pp 182–185.

[30] FER-ZPR, Sortko website, 2012, From http://nihao.fer.hr/sortko

(alternate url http://161.53.18.12/Sortko).

[31] FER-ZPR, Sortko Android market web page, 2012, From https://

market.android.com/details?id¼fer.sortko.com.

SORTING WITH MOBILE DEVICES 9

BIOGRAPHIES

Dr. Ivica Boticki is a researcher and lecturer

in the Department of Applied Computing,

Faculty of Electrical Engineering and Comput-

ing, University of Zagreb, Croatia. His

research interests include mobile learning,

software engineering and engineering educa-

tion. He teaches courses in software engineer-

ing and does research in technology enhanced

learning focusing on two main areas: primary

school education and the enhancement of edu-

cational practices in software engineering education. He is an author

and co-author and reviewer of publications in several journals and

conferences.

Ante Barisic got his master’s degree in com-

puter science and is currently pursuing another

master’s degree in economics at the University

of Zagreb, Croatia. Passionate about product

design and development, he started his career

with developing ICT products in telecom

industry. His main area of interest is the devel-

opment of web and mobile applications.

Dr. Sergio Martin is an assistant professor

and researcher in the Electrical and Computer

Engineering Department at UNED (Spanish

University for Distance Education) since 2006.

He has a PhD in Computer Engineering. He

has participated since 2002 in national and

international research projects.

Neven Drljevic is currently pursuing his mas-

ter’s degree in computing at the University of

Zagreb. Enthusiastic about new technologies

and opportunities they open up for young

people, several years ago he started the Web

Start Contest, a successful local competition

and educational course in developing student

web start-up projects, organized by the student

association eSTUDENT.

10 BOTICKI ET AL.