CSTutor: A Sketch-Based Tool for Visualizing Data Structures · the user edits the code, the diagram will animate and update itself. The data structures represented in CSTutor (Linked

�

�

�

�

�

�

�

�

CSTutor: A Sketch-Based Tool for Visualizing Data Structures

SARAH BUCHANAN and JOSEPH J. LAVIOLA JR., University of Central Florida

We present CSTutor, a sketch-based interface designed to help students understand data structures, specif-ically Linked Lists, Binary Search Trees, AVL Trees, and Heaps. CSTutor creates an environment thatseamlessly combines a user’s sketched diagram and code. In each of these data structure modes, the usercan naturally sketch a data structure on the canvas just as they would on a white board. CSTutor analyzesthe user’s diagrams in real time, and automatically generates code in a separate code view to reflect anychanges the user has made. Additionally, the code can also be edited and any new code changes will animatethe data structure drawn on the canvas. The connection between the data structure drawn on the canvasand the code implementation is intended to bridge the gap between the conceptual diagram of a data struc-ture and the actual implementation. We also present the results of two semester-long studies using CSTutorin a CS1 course. The results indicate that students preferred CSTutor and were more engaged using it thana standard whiteboard lecture; however, results were mixed in quiz and exam performance.

Categories and Subject Descriptors: K.3.1 [Computers and Education]: Computer Uses in Education—Computer-assisted instruction (CAI)

General Terms: Algorithms, Human Factors, Experimentation

Additional Key Words and Phrases: Sketch-based input, data structures, visualization, animation

ACM Reference Format:Buchanan, S. and Laviola Jr., J. J. 2014. CSTutor: A sketch-based tool for visualizing data structures. ACMTrans. Comput. Educ. 14, 1, Article 3 (March 2014), 28 pages.DOI:http://dx.doi.org/10.1145/2535909

1. INTRODUCTION

Instructors in Computer Science traditionally present new data structures usingPowerpoint presentations or drawings on a whiteboard combined with pseudo-code.Usually an instructor will have to draw several instances of a diagram to explain anoperation. This can be troublesome for multiple reasons. First, students may not un-derstand which piece of pseudo-code pertains to which change in the diagram. Second,it is often time consuming and difficult for instructors to draw each instance of the di-agram. We designed CSTutor (see Figure 1) with the goal of bridging the gap betweenthe conceptual diagram of a data structure and the actual implementation.

In addition to diagrams, algorithm visualization tools are used by students and in-structors as a supplemental learning aid. However, studies have shown that algorithmvisualization alone does not improve a student’s level of learning; it is the level of stu-dent engagement with the algorithm visualization that is effective [Naps et al. 2002].CSTutor facilitates the easy creation of diagrams and animations as well as providingan engaging environment with its intuitive pen-based interface.

This work is supported in part by the National Science Foundation, under the NSF CAREER award IIS-0845921 and NSF awards IIS-0856045 and CCF-1012056.Authors’ address: S. Buchanan and J. J. Laviola Jr., Electrical Engineering and Computer Science Depart-ment, University of Central Florida; email: [email protected] to make digital or hard copies of all or part of this work for personal or classroom use is grantedwithout fee provided that copies are not made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page. Copyrights for components of this work ownedby others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.c© 2014 ACM 1946-6226/2014/03-ART3 $15.00DOI:http://dx.doi.org/10.1145/2535909

ACM Transactions on Computing Education, Vol. 14, No. 1, Article 3, Publication date: March 2014.

�

�

�

�

�

�

�

�

3:2 S. Buchanan and J. J. Laviola Jr.

Fig. 1. A screenshot of the sketching area in Linked List mode after nodes 29 and 3 are added.

The intended users of CSTutor are the instructor and students in a beginning datastructures and algorithms class (called CS1 at the University of Central Florida). Thesystem lets the user sketch elements of a data structure and perform operations onthem through the recognition of handwritten symbols and gestures. Once a gestureis recognized the operation on the data structure is animated while simultaneouslyadding automatically generated code to the code view. In addition, the code can beedited and any new code changes will animate the drawing in real time. In other words,if the user sketches changes to the diagram, the code will dynamically change, and ifthe user edits the code, the diagram will animate and update itself. The data structuresrepresented in CSTutor (Linked List, Doubly Linked List, Binary Search Tree, Heap,and AVL Tree) were chosen specifically because they are the most basic data structurestaught in introductory computer science classes.

We chose a pen-based interface for CSTutor’s sketching area so that it would closelyresemble lectures in class on a whiteboard, responses to exam problems with pen-cil and paper, and brainstorming sketches, when solving programming assignments.Oviatt et al. [2012] explored the impact of interface affordances on human ideation,problem solving, and inferential reasoning, and found if interface affordances are wellmatched with a task domain, they can increase physical or communicative activitypatterns that facilitate creating new mental models. Oviatt and Cohen [2010] alsoshowed that when students use a poorly matched interface, such as a keyboard-basedinterface for certain STEM tasks, they experience a higher cognitive load, which canundermine their performance. Specifically, during science activities, students who useda keyboard-based graphical interface solved problems more slowly, solved fewer prob-lems correctly, and forgot more problem content, compared with those using a digitalpen interface. Based on these findings in affordance theory, we designed the CSTutorenvironment to foster exploration and learning by allowing users to easily sketch theirown examples, see the data structures react, and experiment with different operations.CSTutor provides an interface that is appropriately mapped to the task domain, witha pen interface provided for sketching and manipulating diagrams, and a keyboardinterface for entering and editing code.

In order to gauge the usefulness of CSTutor in aiding CS1 students, we conductedtwo semester-long studies; the first in Fall 2011 and the second in Spring 2012. In the


�

�

�

�

�

�

�

�

CSTutor: A Sketch-Based Tool for Visualizing Data Structures 3:3

first semester study, the treatment group was taught using CSTutor on the projector inthe lab sessions, while the control group was taught with a traditional whiteboard lec-ture. In the second semester study, the first treatment group was taught using CSTutoron the projector in the lab sessions as before, in the second treatment group the stu-dents used CSTutor during the lab sessions, and the control group was taught witha traditional whiteboard lecture. A video demonstrating how CSTutor can be used toteach data structures can be found online.1

We begin, in Section 2, by discussing work related to CSTutor. In Section 3, we de-scribe the CSTutor user interface and the interaction paradigms for each data struc-ture. Section 4 explains the implementation of the four primary components of theCSTutor architecture: the Ink Handler, the Code Handler, the Data Structure Han-dler, and the Animation Handler. We then present two semester-long experimentalevaluations of CSTutor and the results, in Sections 5 and 6. This is followed by a dis-cussion of the experimental results, limitations, and future work in Section 7. Finally,we present conclusions in Section 8.

2. RELATED WORK

2.1. Algorithm Visualization Systems

Many algorithm visualization (AV) and animation tools have been developed over thepast 30 years, without gaining much success in the classroom. However, it has beenshown that appropriate animations are most pedagogically useful when they supportstudent interaction and active learning [Naps et al. 2002]. We present an overview ofalgorithm visualization systems, and focus on more recent systems that incorporatestudent interaction.

In early systems, such as BALSA, Tango, POLKA, and XTANGO, users created vi-sualizations by inserting calls to the animation system directly into the source code tobe visualized [Brown and Sedgewick 1985; Lawrence et al. 1994; Stasko 1990, 1992].ANIMAL also allowed general-purpose animations to be constructed via its built-inlanguage, ANIMALSCRIPT [Roßling and Freisleben 2001]. These systems allowed so-phisticated visualizations to be produced; however, the learning curve was steep andnot ideal for new students. MatrixPro takes an alternative approach by allowing stu-dents to easily create animations by graphically invoking operations, yet there is nocoding component [Karavirta et al. 2004]. The goal of CSTutor is to not require thestudent to know anything about the visualization creation process, but to code andsketch data structure diagrams the way they normally would but still be able to playanimations by setting a break point in their code.

Two of the most widely used software visualization tools today are JGrasp and Jeliot,which focus heavily on debugging code rather than understanding the concepts of datastructures. JGrasp is a comprehensive IDE that generates visualizations as neededfrom the user’s actual program during routine development [Cross et al. 2007]. JGraspis similar in scope to CSTutor in that it focuses on core data structures such as treesand linked lists. Although it focuses on creating visualizations from the user’s codelike CSTutor, it does not allow the user to edit or create code from the diagram. Jeliot[Moreno et al. 2004] is similar to JGrasp, however it focuses on beginning conceptssuch as expression evaluation and assignment of variables.

JHAVEPOP [Furcy 2009] based on the JHAVE framework [Naps et al. 2000], is an-other teaching tool that focuses solely on linked lists by presenting specific linked list

1http://youtu.be/uEja cxrGCg


�

�

�

�

�

�

�

�


problems to solve, and then graphically displaying their results. JHAVE is a network-based framework in which animation systems can be embedded. Many existing AV’shave been created using JHAVE, where the AV is set up as a series of annotated di-agrams in one pane, and pseudocode in the adjacent pane. As the user steps throughthe annotated diagrams, the associated pseudocode is highlighted. There are also ques-tions that pop up for the user to answer. The focus of this program is very similar toCSTutor, since the users can see the diagram changing as the code executes. Howeverwe wanted to take this idea further by allowing students to experiment and solve theirown problems without using canned exercises by allowing the generation of their owncode and diagrams from scratch.

In addition, JAWAA and TRAKLA2 are algorithm visualization tools that have hadpositive results in the classroom. JAWAA is a Web-based algorithm animation script-ing language that allows students to print animation commands from their own pro-grams [Akingbade et al. 2003]. In contrast to CSTutor, JAWAA requires the studentsto have knowledge of what they want the animation to do, whereas CSTutor createsthe animations behind the scenes and does not require the user to enter any specialprint commands. With TRAKLA2 students can solve algorithm problems with a dragand drop interface and then are provided automatic assessment [Nikander et al. 2004].TRAKLA2 is especially innovative with its use of interactive exercises, where studentscan demonstrate their proficiency by providing information about key events thathappen during algorithm execution. CSTutor differs from TRAKLA2 since it has aninteractive pen-based interface as well as a source code component, allowing more ex-pressive interaction and a link between the visualization and its implementation.

2.2. Algorithm Visualization Evaluations

There are many experimental studies that analyze the educational effectiveness of al-gorithm visualization systems. Hundhausen et al. [2002] conducted a meta study ofAV’s focusing on empirical effectiveness studies. They analyzed 24 experimental stud-ies and each one’s independent variables (the learning theory explored) and dependentvariables (measurement techniques). They concluded that the most successful usesof AV’s are those that are used in a way that actively engage students in the processof learning. Examples of such usage are prediction exercises as used in Byrne et al.[1999], what-if analyses of algorithmic behavior as used in Lawrence [1993], and pro-gramming exercises as used in Kann et al. [1997]. They also concluded that proceduralknowledge serves as a more indicative measure of an AV’s benefits than conceptualknowledge, which was also concluded by Kehoe et al. [2001]. We were able to leveragethese findings in our experimental design by actively engaging students with CSTutorduring class time and using measurements that test procedural knowledge.

Recently, Urquiza-Fuentes and Velazquez-Iturbide [2012] used a similar experi-mental design to ours. They studied the effect on student performance of differenteducational uses of program animations: their construction by students and their ob-servation by students. In addition they compared both approaches to a traditionalteaching methodology, finding mixed results on simple topics versus complex top-ics, and suggested integrating methods. Alternatively, Orsega et al. [2012] evaluatediSketchmate’s utility from the instructor’s perspective. They compared the efficiencyand accuracy of using iSketchmate versus using a typical whiteboard, finding thatit was both more efficient and more accurate. Other work on experimental eval-uations of algorithm visualizations has been done as well, including Stasko et al.[1993], Crosby and Stelovsky [1995], Kann et al. [1997], Byrne et al. [1999], Nikanderet al. [2004], Crescenzi and Nocentini [2007], Hundhausen and Brown [2007], Crosset al. [2009], and Velazquez-Iturbide [2012].


�

�

�

�

�

�

�

�


2.3. Sketch-Based Systems

There have been several sketch-based tools developed for learning and enteringinformation in the computer science domain. SetPad and LogicPad are sketch-based applications that aid students with Boolean algebra and discrete math prob-lems by providing visualizations [Cossairt and LaViola 2012; Kang and LaViola2012]. SketchUML and Tahuti are sketch-based tools for UML class diagrams [Ham-mond and Davis 2006; Qiu 2007]. GraphPad is a Web-based software system thatlets the instructor and students interact electronically during class on Tablet PCs.Students can also submit solutions to graph problems and receive feedback [Pargasand Bryfczynski 2009]. CSTutor is different from GraphPad in that GraphPad doesnot incorporate the code implementation of the data structure and does not useanimation, whereas CSTutor does both. The system presented in Yuan et al. [2008]is a pen-based flowchart recognition system that recognizes flowchart components andgenerates runnable C code. This is similar to CSTutor in that it recognizes ink andthen generates code, however it is not applicable to data structures and does not reflectchanges to the code in the sketched diagram. To the best of our knowledge, CSTutoris the first pen-based application that incorporates animation and coding, designed foraiding Computer Science students in understanding data structures.

3. CSTUTOR PLATFORM AND USER INTERFACE

CSTutor requires a tablet PC running Windows with a stylus and keyboard. TheCSTutor user interface has two main views that users are able to switch between.The default view is a canvas (see Figure 1) that has been designed to look and feellike pen and paper and provides feedback in an unobtrusive way. The buttons on thescreen are minimalistic, and all of the commands are given through gestures wherepossible. Sketch-based gestures are considered predefined motions that indicate a com-mand when recognized: such gestures used in CSTutor are circle, rectangle, line, andscribble-erase. We aim to model the gestures after what is normally done with pen onpaper. For instance, circling a value when sketching a Binary Search Tree node is whatis normally done with pen and paper, thus the completion of the circle gesture is whatalerts the system that a new node has been added to the tree.

Running alongside the canvas is the code view area, which gives the user a scaf-fold since any sketched data structure generates corresponding code, giving studentsa head start in writing various data structure operations. The code view behaves justlike a typical IDE by highlighting keywords and providing compile and debug informa-tion. However what sets CSTutor apart from a typical compiler is that any updates tothe code will actually impact the data structure in the canvas, and the user can stepthrough the animation to see how their code impacts the data structure.

3.1. Sketching Area

The user is able to choose between the different data structures, which allows us tolimit the available gestures to only what operations are available for any given datastructure. In order to keep the UI as natural as possible, a combination of severalbasic gestures are used for each interaction mode. Linked Lists and Doubly LinkedLists use a combination of rectangle, line, and scribble erase gestures. For the trees(Binary Search Tree, Heap, and AVL Tree) a combination of circle, arrow, tap, andscribble erase gestures are used. All of the gestural operations that we have chosenare based on how a professor would teach these concepts using a diagram, and thecontent comes from a variety of course lectures and textbooks [Standish 1995].

Additional pointers to nodes are represented as blue rectangles on the canvas. Thehead pointer remains static for each different data structure to emphasize that it is


�

�

�

�

�

�

�

�


Fig. 2. The code area in Linked List mode after nodes 29 and 3 are added to the list.

necessary to have a pointer to the beginning. The temp pointer box is also alwaysthere as a temporary pointer to any new nodes, or nodes that are being modified. Theuser can use a line gesture to connect any of these pointers to different nodes. Forexample, changing which node the head points to is useful for inserting a node to thefront of a linked list, and changing which node the temp pointer is pointed to can beuseful for deleting a node in these data structures. In addition, any pointers the usercreates in their code appear on the canvas as blue rectangles with the correspondingvariable names.

3.2. Code Area

Any sketch created on the canvas always creates a section of code in order to showthe connection between the diagram and any code changes. When a sketch is com-pleted, the corresponding code is simultaneously added to the main method in the codearea (see Figure 2) and is also shown in an information box in the user’s sketch area(see Figure 1). The auto-generated code is meant to demonstrate how memory works,and is not the modular code you would see in a program. Users can edit the auto-generated code, add their own functions and operations, compile and run the code, andstep through the resulting changes to the diagram. Compiler output is also displayedin the user’s code area, and any warning or error messages are shown.

The user can create their own animations by writing functions in the code area.The user then sets a break point at the line of code where they want to begin theanimation from. The Show Me button allows the user to step through each line of code(highlighted in the code area) and to see how their diagram changes at each step. Eachstep in the animation has the potential to create new nodes or pointers, delete existingnodes, or move the connections of any of the nodes or pointers.

3.3. Interaction Modes

3.3.1. Linked List and Doubly Linked List. The sketching area is designed to allow the userto do the same operations on the data structure through sketching as they can through


�

�

�

�

�

�

�

�


Fig. 3. The steps required to delete a node from a Doubly Linked List: (1) Assign temp to the node to bedeleted, (2) bypass the node to be deleted, (3) free the node pointed to by temp.

code. They can create nodes, connect and disconnect nodes, assign/un-assign pointersto nodes, delete nodes and pointers, and update the data and next values. For example,a linked list node is created by writing the value of the node and then boxing that re-gion in with a rectangle. When the rectangle gesture is completed, the surroundedstrokes are sent to an online recognizer that analyzes the user’s handwriting to deter-mine the numerical value. After the node is drawn it is translated and scaled to makeroom for subsequent sketches. Then, a small box is drawn within the node to representthe allocated memory for a pointer to the next node. A null pointer stroke is also addedto this memory field for nodes where the next link is null (see Figure 1). Sketching anode on the canvas also affects the code in the code window, and generates code thatdoes the following: (1) allocates a node in memory, (2) assigns a value to that node, and(3) sets the node’s next value to null.

For operations such as delete, instead of the user simply scratching out a node werequire that the user goes through each step they would have to implement in codeusing gestures on the canvas (see Figure 3). For example, the temp pointer first needsto be assigned to the node being deleted by drawing a line to the node from the temppointer box. Secondly, the neighboring nodes need to be linked to each other to bypassthe deleted node. Finally, the node pointed to by temp can be freed by using a scratch


�

�

�

�

�

�

�

�


Fig. 4. The sketching area in Binary Search Tree mode. Node 5 was added to the correct location and iscolored green, whereas node 100 was added to the incorrect location and it is colored red.

out gesture on the temp node and the node is deleted from memory. These steps wouldcreate the following code:

temp = head->next;head->next = head->next->next;head->next->next->prev = head;free(temp);

We designed the Linked List and Doubly Linked List interaction modes to requirethat the lists be connected, or at least that all nodes are accessible by a pointer, todemonstrate memory leaks to the user. For example, if a user creates their first nodewithout connecting it to the temp or head pointer, any future nodes will not be addedand the user is shown a warning message. Additionally, in the preceding delete exam-ple, if the user bypasses the node they wish to delete without first assigning temp tothat node, then temp is automatically assigned to that node. This rule is presentedin Standish [1995], and says that access must be preserved to all nodes that will beneeded later in the solution of a problem. Similarly, the user cannot scribble erase anode that does not have a pointer already assigned to it (as is the case in code forneeding a reference to delete a node), thus the scribble erase gesture would do nothingin that case. There is no specific gesture for iteration, yet the user can iterate to anynode in the list by sketching a straight line from any pointer to the node in the list.The auto-generated code then fills in the iteration code required to get to the first nodewith the specified value.

3.3.2. Binary Search Trees. Tree nodes are created by writing the value of the node andthen circling that number, which sends the surrounded strokes to an online recognizer,just as with the Linked List mode. After the node is drawn it is dynamically arrangedon the canvas and lines are drawn to each child node if the children are not null (seeFigure 4). Similar to the Linked List mode, sketching a new node with a value of 5creates the following code:

temp =(struct node*)malloc(sizeof(struct node));temp->data = 5;temp->left = NULL;temp->right = NULL;head = insert(head, temp);


�

�

�

�

�

�

�

�


Fig. 5. The sketching area in heap mode. Node 16 has just been added to the heap, and the up arrowgesture has been performed to swap 16 with its parent node. Beneath the heap the current state of thearray is shown.

Note that since inserting into the tree requires more than a single line of code, wehave abstracted the insert code into its own function and we hide it from the user’scode window. We chose to hide the implementation of insert from the user so that theycan code their own insert function and visualize it on the canvas.

We initially supported having a newly created node automatically moved to the cor-rect position in the tree. However, we want to ensure that students know how to inserta node according to the binary search tree property, so we make sure the user sketchesthe node in the proper area of the tree. If the node is added to the correct position, it iscolored green. Otherwise, if the node is in an incorrect position the strokes are coloredred and the user cannot make additional operations on the tree until the offendingnode is scribbled out (see Figure 4).

In addition to inserting new nodes in the tree, the user can delete a node by scratch-ing out the node. The user can scratch out any node in the tree and the system will callthe delete function on that node’s value, then the delete function will internally searchthe tree for that value. Since insertion and deletion are the main operations made toa binary search tree, gestural operations (e.g., circle for creating nodes and scribbleerase for deleting nodes) are provided for them and any other operations on the treemust be implemented in the user’s code.

3.3.3. Heaps. The heap data structure we support is a minimum binary heap, sincethis data structure is taught in entry level Computer Science courses. The heap isdisplayed in the same manner as the Binary Search Tree, with an additional arrayunderneath that is updated after each operation. Although a binary heap is usuallyvisualized as a tree, it is generally implemented using an array [Standish 1995], thusit is necessary to show the heap and array at the same time (see Figure 5).

The main operations performed on a heap are inserting new nodes and removingthe minimum element. Since there are many steps for the insert and remove mini-mum operations, we have abstracted both into their own functions, just like the insertoperation in Binary Search Tree mode.

Similar to the Binary Search Tree mode, if the node is added to the correct spot (thenext open leaf node), the node is colored green, otherwise it is red. If the node that isinserted is less than its parent, then a percolate up function is recursively called, sothat the node can continue to swap locations with its parent until it is in its correct


�

�

�

�

�

�

�

�


location in the tree. Originally, the swap operations were animated immediately, sothe node would be swapped up the tree without any user input. However, since manyexams ask the student to show each step in an insert heap operation, we make theuser draw an up arrow gesture for each necessary swap up, and a down arrow gesturefor each swap down (see Figure 5).

The second operation supported in heap mode is to remove the minimum node, whichis done with a scribble erase gesture on the root of the tree. Once this node is removed,it must be replaced with the last leaf node in the tree. Since we want to ensure thatthe user knows which node to select as a replacement, we require a tap gesture next tothe replacement node. If the wrong node is selected it is colored red, and the programwill wait until the correct node is selected before finally animating the swap. Oncethe root is replaced, it must be swapped with its children until the root becomes theminimum element in the tree if it is not already. After each operation on the heap,the array underneath the tree is updated to reflect the current state of the heap datastructure as it exists in memory. In the future we would like to add the ability to editthe array in the sketching area as well as the tree, but at this time that functionalityis not supported.

3.3.4. AVL Trees. Many students struggle with understanding the conceptual opera-tions on AVL trees as well as their implementation. One common approach to teachingAVL trees is to teach the conceptual operations on an AVL tree diagram before delv-ing into the code implementation. Some examples of conceptual operations that aretaught are identifying the balance factor, determining if a rotation is needed, deter-mining what kind of rotation is needed, and finally what the tree should look likeafter it has been rotated. In a classroom setting all of these operations are initiallypresented without the code and are still difficult to understand. Once the code is pre-sented many students become even more confused with the recursive implementation.CSTutor adds guidance to help with understanding conceptual operations and delvinginto the code.

When a user is sketching an AVL tree in CSTutor and a rotation is needed, theuser cannot continue sketching until the rotation is completed. A single rotation iscompleted with an arrow gesture in the required direction, and once the correct gestureis done the tree animates to reflect the changes of the rotation. Since a double rotationis simply a call to two consecutive single rotations, we enforce that the user makes twoarrow gestures for a double rotation and the user can visualize each single rotationseparately to see specifically what the code is doing. We designed CSTutor to forcestudents to actively rotate the tree with arrow gestures, instead of allowing users topassively observe the AVL tree animations, to promote active learning.

4. ARCHITECTURE

The software architecture of CSTutor is divided into four primary components: thesketching area (the Ink Handler), the code area (the Code Handler), the Data Structurecomponent (the DataStructure Handler) and the animation component (the AnimationHandler). The components of CSTutor are shown in Figure 6.

4.1. Ink Handler

The Ink Handler deals directly with the user input, and it consists of a handwritingrecognizer and a gesture recognizer. This handler directly interfaces with the DataStructure Handler and the Animation Handler by differentiating among the differ-ent operations and gestures. The sketching area for CSTutor is built on Microsoft’sInkCanvas, which is a Windows Presentation Foundation library that receives anddisplays ink strokes. We implemented gesture and character recognition to process the


�

�

�

�

�

�

�

�


Fig. 6. CSTutor component relationship diagram.

ink strokes collected. Gestures are detected by using the iStraw algorithm [Xiong andLaViola 2009] for cusp (or corner) detection in combination with heuristics definingeach gesture. We found this to be a significant improvement over Microsoft’s built-ingesture recognizer, which was not only inaccurate at times but also limited us using apredefined gesture set that was not appropriate for our application. For handwritingrecognition, we chose to use the $N Recognizer [Anthony and Wobbrock 2010] becauseit is capable of accurately identifying multi-stroke symbols. Gestures are used to insertnodes, delete nodes/strokes, connect/disconnect nodes, and move nodes. The Ink Han-dler interacts with the Animation Handler only when it is waiting on a gesture fromthe user to continue animating (such as when the heap is waiting for an arrow gesturefrom the user), otherwise the gesture is an operation on the data structure and is sentto the Data Structure Handler.

4.2. Code Handler

4.2.1. Compiling and Running Code. The Code Handler compiles the user’s code and out-puts any compile-time errors to the user via a compiler output window in the codearea. If the user’s code compiles without errors, it is scanned for changed lines so thatonly the code that has been recently modified is animated (otherwise the user wouldhave to rewatch old animations). The Code Handler only scans the main method forcode changes, since functions will only be seen if they are called from those changedlines. The Code Handler works by doing two passes through the user’s code. Duringthe first pass a list of function names is created to quickly determine if a function iscalled. Once the changed lines of code have been determined, the second pass dividesthe changed code into logical chunks such as declaration, assignment, loop start/end,and conditional start/end. The second pass also determines which functions have beencalled, and will analyze the body of the relevant functions. Statements are then in-serted after each logical code chunk to print the current state of each pointer and therelevant operation into our own Data Structure XML format.


�

�

�

�

�

�

�

�


After the user’s code has been analyzed, a new version of their program is cre-ated that contains the injected XML print statements. This code is compiled and runthrough the gcc command line compiler, and at runtime, the Data Structure steps areprinted to an XML file. This XML file is then read by the Data Structure Handler whenthe Show Me button is pressed. In the event that the user’s program crashes, a partialanimation list is created so that they can trace the error.

4.3. Data Structure Handler

The Data Structure Handler contains a fully implemented version of each data struc-ture. This Handler also keeps track of the current and previous states of the datastructure, the strokes associated with each element, and the position of its nodes onthe screen. If any operations are made to the data structure either by code or by ges-tures, the current and previous states of the data structure are modified so that theuser can replay an animation. The final task of the Data Structure Handler is to sendoff animations to be queued in the Animation Handler.

4.3.1. Updating a Data Structure in the Code Area. When the Show Me button is pressed,the Data Structure Handler reads the next step from the XML file. Each XML stepis interpreted as an operation on the current data structure, and if the current datastructure is modified, changes are passed to the Animation Handler to create an an-imation. If a new node is generated through the user’s code we preserve the feel ofhandwritten numbers by creating an artificial node from the training data of a userso that a new node appears in their handwriting on the canvas. Once the final XMLstep is shown, the XML file is cleared, and the user will have to recompile their codeto repeat the animations.

4.3.2. Updating a Data Structure in the Sketching Area. Updating a data structure withthe sketch-based interface begins by either erasing an ink stroke, connecting nodes orpointers with a line, or creating a node with a circle or rectangle gesture.

When the Ink Handler gesture recognizer detects a scribble-erase gesture, the scrib-ble stroke is removed from the canvas and sent to the Data Structure Handler. TheData Structure Handler then tests to see which nodes or pointers the scribble strokeintersects with. If the scribble stroke intersects with part of a node container stroke(the surrounding rectangle stroke for Linked Lists and Doubly Linked Lists or circlefor Binary Search Tree and Heap), then the node and all of its outgoing pointers (next,previous, left, or right) are removed from the canvas and the data structure. If thestrokes are instead part of an arrow pointer (whether extending from a data structurenode or a pointer), the corresponding pointer is removed from the canvas and replacedwith a null pointer. In addition, the data structure object’s pointer is set to null.

Alternatively, when the Ink Handler gesture recognizer detects a line gesture, theline stroke is sent to the Data Structure Handler. The Data Structure Handler thenperforms an intersection test with each of the data structure nodes and pointers. Ifa valid connection is found, the canvas and the data structure object are updated toreflect this connection. Whether a valid or invalid connection is found, the line strokeis removed from the canvas by the Data Structure Handler.

After each operation that is detected in the ink canvas, an auto-generated block ofcode is added to the code window so that the code provides an accurate representationof the diagram on the canvas. It is also possible that incorrectly generated code isadded to the canvas if the steps are not in the correct order—we leave discovering thisup to the user as part of the learning process.


�

�

�

�

�

�

�

�


4.4. Animation Handler

The Animation Handler receives instructions from the Data Structure handler interms of an updated set of node positions, as well as a list of all the current arrowsand pointers on the screen. The Handler then uses this information to intelligentlyqueue the animations sequentially by each operation. Each animation in the queue isplayed on a timer, and if a gesture is required it waits until the gesture is completedby the user.

5. FIRST EXPERIMENTAL EVALUATION

5.1. Research Hypotheses

After developing CSTutor we conducted an initial assessment by demonstrating it toa CS1 class of 88 students as a review before their final exam [Buchanan et al. 2012].The majority of survey responses were positive when rating the perceived usefulnessof CSTutor in their CS1 course. We then wanted to explore whether CSTutor had anyeffect on student’s comprehension and engagement, which led us to the first of twosemester-long experimental evaluations covered in this section. The primary researchhypotheses for the first experiment are the following.

H1. The CSTutor group will perform better than the Control group on weekly labquizzes.H2. The CSTutor group will perform better than the Control group on courseexams.H3. Students will prefer CSTutor to a standard whiteboard lecture.

Essentially, we postulate that student’s increased engagement with the dynamic na-ture of CSTutor as well as seeing the connection between the diagram and the codechanges will allow the students to comprehend the material better.

5.2. Subjects and Apparatus

The first experiment with CSTutor took place during the Fall 2011 semester in a CS1course that taught beginning data structures and algorithms. The class originally con-sisted of 267 students, however at the end of the semester 190 students remained,which is expected, since the expected number of students to drop this course is high.The between-subjects research design used a mixed method approach involving quan-titative and qualitative data. The class was randomly divided into 6 lab sections thatmet once a week, where 3 of the labs were assigned the control treatment and 3 ofthe labs were assigned the experimental treatment. The difference between the con-trol and treatment groups was that the treatment labs were given a lecture using theCSTutor tool running on an HP tc4400 Tablet PC connected to a projector, and thecontrol labs were given a lecture on a standard whiteboard. Each student attended alab session once a week for a total of 13 labs. In 7 of these labs an experiment wasconducted, in the remaining 6 labs there was a standard whiteboard lecture and aworksheet was given to all students. All labs were taught by the same instructor. Theexperiments covered the following topics: Linked Lists Intro, Linked Lists Advanced,Binary Search Trees Intro, Binary Search Trees Advanced, Sorting, Heaps, and AVLTrees.

5.3. Experimental Design and Procedure

All students received the same three lectures each week, but attended one smaller labclass that was meant to reinforce and practice the concepts taught during the lectures.The labs were 55 minutes long. The first 10 minutes were spent doing an individualpretest on the concepts to be taught during lab that day. During the next 30 minutes


�

�

�

�

�

�

�

�


the instructor gave a participatory lecture, where either the lecture was given usingthe CSTutor tool or on a standard whiteboard. Finally, during the last 15 minutes thestudents were given a quiz.

5.4. Metrics

The pretests and quizzes were designed so that the pretests contained one to threequestions, and the quiz would have multiple questions but would also contain a vari-ation of the pretest question(s). Variations of the pretest questions were duplicatedon the quiz so that we could calculate gain scores. Similar approaches using a pretestand post-test were used in Bennedsen and Schulte [2010] and Grissom et al. [2003].The quiz also contained a short Likert scale survey, with an agreement scale rangingfrom strongly disagree (1) to strongly agree (7), and a question for comments. An ex-ample pretest and quiz are shown in Appendix A. The survey questions given to thetreatment group at the end of the quiz were as follows.

(1) CSTutor is helpful in understanding the implementation of Linked Lists.(2) CSTutor helps in visualizing the steps in Linked List functions.(3) CSTutor makes understanding Linked Lists difficult.(4) Other comments about CSTutor:

The alternative survey questions given to the control group were the following.

(1) The lecture was helpful in understanding the implementation of Linked Lists.(2) The lecture helped in visualizing the steps in Linked List functions.(3) The lecture made understanding Linked Lists difficult.(4) Other comments about the lecture:

In addition, the three course exams were used as a comparison measure. Each examwas given at approximately the end of each month of the course, so Exam 1 coveredthe first month’s material, Exam 2 the second month’s material, and the Final Examwas longer and cumulative, covering the entire course’s material.

5.5. Results

5.5.1. Quantitative. Since the students had not been exposed to CSTutor before thefirst exam it was used as a baseline measure to see if the groups of students performedequally well initially. An independent samples t-test was performed on the Exam 1scores showing no significant difference between groups (t220 = −3.471, p = 0.69),which provides some evidence that the groups were initially equal. Exam 2 and theFinal Exam were used to compare the groups’ performances. The results are shownin Table I; there was significance between group performance for the Final Exam butnot for Exam 2. The treatment group did significantly better on the final exam; theaverage in the control group was 63.7, the average in the treatment was 70.8 (t178 =1.969, p < 0.05). These results indicate that the CSTutor group had better retention ofthe topics covered over the entire course.

We also compared the students’ weekly performance on the pretests and quizzesusing an independent samples t-test. For the most part, the raw quiz scores and gainscores were not significantly different between groups. However, during the third weekof treatment the control group performed significantly better on the Binary SearchTrees Intro (BST 1) lab. The average gain in the control group was 3.5, the averagegain in the treatment group was 2.0 (t178 = −3.471, p < 0.01). During the BST 1 labthe instructor observed that many students really enjoyed the Binary Search Treeanimations. It is possible that the students were distracted by the animation or theanimation went too fast.


�

�

�

�

�

�

�

�


Table I. Experiment 1—Quiz and Exam Results

The quiz results show the gain or the difference between the pretest and quiz scores. There werealso two exams that assessed the material taught in the experimental labs; Exam 2 coveredLinked Lists through Binary Trees and the Final Exam covered all material. The CSTutorgroup had a significantly higher Final Exam average score and a significantly lower Quiz gainfor Binary Search Trees Intro.

Treatment - Control -Lab Topics Average Gain Average Gain Independent T-Test

Linked List Intro 1.65 1.56 t219 = 0.42, p = 0.67Linked List Advanced 1.45 1.03 t252 = 1.458, p = 0.15Binary Search Trees Intro 1.99 3.53 t178 = −3.471, p < 0.01Binary Search Trees Advanced 2.37 2.11 t169 = 0.873, p = 0.37Sorting 2.46 2.75 t158 = −0.866, p = 0.39Heaps 1.20 1.33 t165 = −0.350, p = 0.72AVL Trees 1.17 1.51 t153 = −0.988, p = 0.33

Treatment - Control -Exams Average Average Independent T-Test

Exam 2 78.4 80.4 t187 = −0.869, p = 0.39Final Exam 70.8 63.7 t178 = 1.969, p < 0.05

At the end of each quiz there was a short survey that contained three 7-point LikertScale questions. The three questions in the survey are shown at the end of the quizshown in Appendix A Table, IX. The responses to the survey questions were comparedbetween the treatment group and the control group using the Mann-Whitney U Testand the results are shown in Table II. The survey responses were significantly betterfor the CSTutor group for the Linked List Introduction, Linked List Advanced, Sort-ing, and AVL Trees topics, and there was no significant difference for other topics.We believe that the students rated these topics higher for a number of reasons. Of-ten, Linked Lists are the first abstract data type that these students have seen in thecourse and as such struggle with it. Thus students were excited to see the dynamicconnection between the code and the diagram and visualize different concepts in ac-tion. For the sorting lab, seeing different quicksort and merge sort problems in actionhelped demonstrate the conceptual algorithms. Finally, many students struggled withidentifying unbalanced AVL trees as well as how to rebalance them. Thus being ableto see different examples in action as well as to ask “What if we did this?” and thenhave the instructor demonstrate their examples on the fly was helpful.

5.5.2. Qualitative. We read the students’ comments in the weekly surveys and clus-tered their responses into general categories according to the themes that we ob-served. The categories were: Like, Like but buggy, Like specifically the visuals, Want touse, How does it work?, Prefer Powerpoint, Confused, Improvements needed, Did notlike, Neutral. The distributions of comments across the whole semester are shown inTable III. The distribution of comments from the Linked List Intro lab are shown inFigure 7. There were 74.6% positive comments, 23.7% negative comments, and 1.7%neutral comments, with the preference for a visual way of learning as the majority ofcomments at 33.9%. The idea that the students enjoyed a visual way of learning arosefrequently in their answers to the open-ended questions. In the open-ended commentsquestion, one student wrote the following.

“I like the visualization of Linked Lists and how it makes stepping throughthe info faster (drawing to auto generated code). The code is clear on the


�

�

�

�

�

�

�

�


Table II. Experiment 1—Weekly Survey Results

There were 3 7-point Likert scale survey questions (Q1–Q3) given at the end ofeach quiz (shown at the end of Appendix A), which asked students to rate theirexperience with CSTutor/whiteboard lectures for each topic.

Treatment - Control -Lab Topics Avg Avg Z P value

Linked List IntroQ1 5.47 (σ= 1.4) 5.29 (σ= 1.4) −1.06 0.29Q2 6.01 (σ= 1.3) 5.66 (σ= 1.4) −2.21 <0.05Q3 5.47 (σ= 1.4) 5.47 (σ= 1.4) −1.71 0.09

Linked List AdvancedQ1 5.88 (σ= 1.3) 5.89 (σ= 1.1) −0.56 0.58Q2 6.12 (σ= 1.1) 6.11 (σ= 1.0) −0.39 0.70Q3 1.81 (σ= 1.3) 2.19 (σ= 1.3) −2.80 <0.05

BST IntroQ1 6.22 (σ= 1.0) 6.19 (σ= 1.0) −0.05 0.96Q2 6.35 (σ= 1.0) 6.40 (σ= 0.9) −0.16 0.88Q3 1.53 (σ= 1.1) 1.61 (σ= 0.9) −1.27 0.21

BST AdvancedQ1 6.05 (σ= 1.2) 6.04 (σ= 1.0) −0.69 0.49Q2 6.20 (σ= 1.2) 6.07 (σ= 1.0) −1.37 0.17Q3 1.65 (σ= 1.2) 1.70 (σ= 1.0) −1.12 0.26

SortingQ1 5.94 (σ= 1.2) 5.40 (σ= 1.6) −1.98 <0.05Q2 6.04 (σ= 1.2) 5.40 (σ= 1.6) −2.30 <0.05Q3 1.82 (σ= 1.2) 2.22 (σ= 1.7) −1.02 0.31

HeapsQ1 6.09 (σ= 0.9) 6.24 (σ= 1.2) −1.72 0.09Q2 6.33 (σ= 0.9) 6.36 (σ= 1.1) −0.84 0.40Q3 1.47 (σ= 0.9) 2.22 (σ= 1.0) −0.17 0.87

AVL TreesQ1 6.35 (σ= 0.9) 6.04 (σ= 1.2) −1.38 0.17Q2 6.51 (σ= 0.8) 6.12 (σ= 1.2) −2.15 <0.05Q3 1.61 (σ= 1.1) 1.81 (σ= 1.3) −0.96 0.34

screen, and I thought it was convenient when someone referenced a linenumber we could all immediately see what he was talking about.”

A few other students commented that they specifically liked the visualization of point-ers and how they change memory locations, such as the following two comments.

“I liked the visual representation of nodes and pointers made it easy to keeptrack of where everything was.”

“Great visuals, would like to use again for pointer actions.”

In addition, many students were curious whether they could download it to playwith or how everything worked. Some students may benefit from the incorporation ofnew technology in the classroom as inspiration for what they can make in the future.An example of an inquisitive student is in the following quote.

“Who made CSTutor? How does gesture recognition work? What was itcoded in? How does it parse and format code?”


�

�

�

�

�

�

�

�


Table III. Experiment 1—Distribution of Open-Ended Survey Question Responses

The comments from each weekly survey were clustered into positive andnegative groups to get an overall feel for student’s reactions to CSTutoron a weekly basis.

Positive Negative NeutralLab Topics Comments Comments Comments

Linked List Intro 74.6% 23.7% 1.7%Linked List Advanced 77.1% 20.0% 2.9%Binary Search Trees Intro 81.5% 14.8% 3.7%Binary Search Trees Advanced 88.0% 12.0% 0.0%Sorting 83.3% 16.7% 0.0%Heaps 86.7% 13.3% 0.0%AVL Trees 88.9% 11.1% 0.0%

Total: 80.3% 18.2% 1.5%

Fig. 7. Experiment 1—Clustered Comments. The graph displays the student’s comments from the LinkedList Intro lab that were clustered into groups. 59 students left comments, 75% were positive, 24% werenegative, 1% were neutral.

The comments remained somewhat similar for the following weeks, but became lessdescriptive as the semester moved on such as “Good”, “Like”, “Nice visuals”, “Preferwhiteboard”. For binary search trees, there were a few that preferred the trees tolinked lists, such as one student who stated “Great for trees, didn’t help me person-ally with linked lists much.” In addition, there were a few students who commentedthat they specifically liked seeing a traversal within a function, such as “It’s interestingto watch it walk through the binary tree, and I find that to be helpful.” In the last weekwhen we covered AVL Tree there were a lot of descriptive and enthusiastic commentsabout CSTutor, such as the following.

“CSTutor was exceptionally helpful with AVL Trees.”

“Visualizing the 4 types of shifts was very helpful!”

“Will we have access to this by semester end? Would be amazing.”


�

�

�

�

�

�

�

�


6. SECOND EXPERIMENTAL EVALUATION

6.1. Research Hypotheses

The first experimental evaluation allowed us to examine whether CSTutor would bebeneficial when the instructor uses it to teach a lab. We also wanted to explore theeffectiveness of CSTutor when the students use the tool themselves in lab. We decidedto have three experimental groups for the second experiment: the Treatment I grouponly has the instructor teach the lab with CSTutor on the projector, the TreatmentII group is the same as the Treatment I group but the students also have tablets oftheir own to follow along and use CSTutor themselves, and the Control group has astandard whiteboard lecture. The research hypotheses for the second experiment arethe following.

H4. The Treatment II group will perform better than the Treatment I group, andboth will perform better than the Control group on the lab quizzes.H5. The Treatment II group will perform better than the Treatment I group, andboth will perform better than the Control group on the course exams.H6. Both the Treatment I group and the Treatment II group will prefer CSTutorto a standard whiteboard lecture.H7. Both the Treatment I group and the Treatment II group will be more engagedby CSTutor than a standard whiteboard lecture.

Essentially, we postulate that students’ increased engagement by using CSTutor andexperiencing the connection between the diagram and the code changes will allow thestudents to comprehend the material better.

6.2. Subjects and Apparatus

The second experiment with CSTutor took place during the Spring 2012 semester inCS1. Originally there were 236 students and after the drop period there were 200students. The second experiment built upon the previous experiment by including asecond treatment group, where the students would use CSTutor themselves to workthrough problems during the lab. This time there were 7 lab sections. The 2 labs inthe Treatment I group were shown CSTutor on the projector during the lab time. The2 labs in the Treatment II group were also shown CSTutor on the projector, but couldfollow along with the lecture and had time to work out problems on their own usingCSTutor on a tablet PC at their desk. The other 3 labs were in the control group andhad a standard whiteboard lecture. The instructor in the Treatment I and II groupsand the students in the Treatment II group both used HP tc4400 Tablet PC’s. Again thestudents were randomly assigned a lab section, and attended the labs once a week fora total of 13 weeks. Due to time constraints and availability of the tablets we decidedto run the intervention in 4 of the 13 weeks for the following topics: Linked List Intro,Linked List Advanced, Binary Search Tree Intro, and Binary Search Tree Advanced.In order to reduce confounding variables, the same instructor as the first experimentwas used for all labs.

6.3. Experimental Design and Procedure

The labs followed the same format as the previous experiment, where the first 10minutes were dedicated to the pretest, the next 30 minutes contained a participatorylecture, and finally during the last 15 minutes the students were given a quiz. Aboutthree times during the lecture students were given 5 minutes to work out a problemon their own. The only difference from the first experiment is that the TreatmentII group worked out the problems on their own tablets, while the Treatment I andControl groups worked out problems using pen and paper. Generally the Treatment II


�

�

�

�

�

�

�

�


Table IV. Study 2—Quiz and Exam Results

The Gain scores are the difference in scores between the pretest and quiz for each week’s lab topic.There were also two exams that assessed the material taught in the experimental labs; Exam 2 coveredLinked Lists and the Final Exam covered all material.

Group I Group II Group ILab Group I - Group II - Control - vs Control vs Control vs Group IITopics Avg Gain Avg Gain Avg Gain P-value P-value P-value

LL I 1.97 1.83 1.51 t171 = 1.82, p = 0.07 t166 = 1.29, p = 0.20 t133 = 0.50, p = 0.62LL II 1.58 2.05 1.06 t171 = 1.20, p = 0.23 t167 = 2.37, p < 0.05 t134 = −1.10, p = 0.27BST I 2.04 1.89 1.27 t172 = 1.84, p = 0.07 t168 = 1.42, p = 0.16 t134 = 0.30, p = 0.76BST II 7.57 6.92 6.73 t172 = 1.05, p = 0.30 t168 = 0.24, p = 0.81 t134 = 0.72, p = 0.48

Group I Group II Group IGroup I - Group II - Control - vs Control vs Control vs Group II

Exams Avg Avg Avg P-value P-value P-value

Exam 2 87.96 86.73 82.67 t139 = 2.00, p < 0.05 t138 = 1.38, p = 0.17 t111 = 0.43, p = 0.67Final 80.20 77.89 78.33 t144 = 0.47, p = 0.64 t144 = −0.11, p = 0.91 t112 = 0.55, p = 0.58

students would use CSTutor to sketch a Linked List or a Binary Search Tree, write afunction, and then check their work by stepping through the function and visualizingthe changes to their data structure. There were 20 tablets for approximately 30 to 40students, thus each tablet was shared between two students.

6.4. Metrics

The same pretest, quiz, and exam formats were used as in the first experiment. How-ever, in the first experiment many students put improvements in the comments sectionof the survey. While the suggested improvements were interesting we wanted to knowif they would like to use CSTutor in the classroom. Thus, we separated the commentssection into two open-ended questions: (1) suggested Improvements for CSTutor; and(2) do you think CSTutor would be useful in Computer Science 1? Why or why not?

6.5. Results

6.5.1. Quantitative. The first exam was used as a baseline measure since it was givenbefore the students were exposed to CSTutor. A one-way ANOVA was used and therewas no significant difference between groups (F2,197 = 0.04, p = 0.69). The gains fromthe pretests and quizzes was compared between groups using an independent samplest-test as shown in Table IV. As in the previous semester, for the most part the raw quizscores, gain scores, and Likert scores were not significantly different between groups.However, the CSTutor-user group had significantly higher gains during week 2 on theLinked List Advanced lab. On the Advanced Linked List quiz, the CSTutor-user grouphad an average gain score of 2.1, the control group had an average gain score of 1.1(t167 = 2.37, p < 0.05). These results indicate that the CSTutor user group had becomefamiliar enough with the Linked List mode using CSTutor to gain benefits from thesystem. We also compared the Exam 2 and Final Exam scores using an IndependentSamples T-Test. The CSTutor-projector group had significantly higher scores than thecontrol group on Exam 2 with CSTutor-projector group having an average Exam 2score of 88.0, the control group had an average score of 82.7 (t139 = 2.00, p < 0.05).However, there was no significant difference between groups on the Final Exam. Onerationale for these results is that Exam 2 focused on the topics covered with CSTutor(Linked Lists and Binary Search Trees), but the final exam had a broader scope.


�

�

�

�

�

�

�

�


Table V. Study 2—Weekly Survey Results

There were 3 7-point Likert scale survey questions (Q1–Q3) given at the end of each quiz (shown at theend of Appendix A), which asked students to rate their experience with CSTutor/whiteboard lecturesfor each topic.

Treatment I - Treatment II - Control - Kruskal-WallisLab Topics Avg Avg Avg Test

Linked List IntroQ1 x = 6.02, σ = 1.14 x = 5.12, σ = 1.76 x = 5.85, σ = 1.44 χ2

2 = 9.36, p < 0.01

Q2 x = 6.10, σ = 1.13 x = 5.21, σ = 1.75 x = 5.92, σ = 1.34 χ22 = 9.87, p < 0.01

Q3 x = 6.45, σ = 0.81 x = 5.54, σ = 1.69 x = 6.08, σ = 1.36 χ22 = 9.69, p < 0.01

Linked List AdvancedQ1 x = 5.76, σ = 1.29 x = 4.78, σ = 1.84 x = 5.74, σ = 1.27 χ2

2 = 11.20, p < 0.01

Q2 x = 5.69, σ = 1.23 x = 5.04, σ = 1.75 x = 5.85, σ = 1.15 χ22 = 6.93, p < 0.01

Q3 x = 5.93, σ = 1.18 x = 5.46, σ = 1.64 x = 6.07, σ = 1.14 χ22 = 4.88, p = 0.087

BST IntroQ1 x = 5.63, σ = 1.45 x = 5.30, σ = 1.46 x = 6.15, σ = 1.16 χ2

2 = 13.74, p < 0.001

Q2 x = 5.70, σ = 1.43 x = 5.38, σ = 1.36 x = 6.13, σ = 1.17 χ22 = 12.08, p < 0.01

Q3 x = 6.00, σ = 1.37 x = 5.90, σ = 1.24 x = 6.28, σ = 1.12 χ22 = 4.63, p = 0.099

BST AdvancedQ1 x = 5.86, σ = 1.32 x = 5.28, σ = 1.49 x = 6.05, σ = 1.04 χ2

2 = 7.10, p < 0.05

Q2 x = 5.93, σ = 1.23 x = 5.47, σ = 1.40 x = 6.02, σ = 1.21 χ22 = 4.55, p = 0.103

Q3 x = 6.14, σ = 1.23 x = 5.81, σ = 1.37 x = 6.17, σ = 1.07 χ22 = 2.06, p = 0.356

At the end of each quiz there was a short survey that used a 7-point Likert Scale.The three questions in the survey are shown at the end of the quiz shown in AppendixA Table IX. The responses to the survey questions were compared between the groupsusing the Kruskal-Wallis Test; the results are shown in Table V. For the Linked ListIntro topic, the Mann-Whitney U test was used to test for significance between pairsof groups. The results indicated that the students answered higher for Q1, Q2, Q3in the CSTutor projector group versus the CSTutor user group (Z = −2.80, p < 0.01,Z = −2.92, p < 0.01, Z = 3.08, p < 0.01 for Q1, Q2, Q3 respectively), and for the con-trol group versus the CSTutor user group (Z = −2.57, p < 0.05, Z = −2.52, p < 0.05,Z = −2.07, p < 0.05). For the other topics, there is no significance for all of Q1–Q3,however where there was significance it indicated lower ratings for the CSTutor usergroup versus the other groups. The results indicate that the students using CSTutorin the lab may have needed more time to get the proper benefits from CSTutor. Thesestudents effectively got 30 minutes with CSTutor per lab which did not allow them touse it in a way that did not feel rushed or be able to experiment and feel comfortablewith it. It was also indicated by these students in their survey responses that they didnot have enough time with CSTutor, as summarized in the next section.

6.5.2. Qualitative. The second experiment had the benefit that the treatment groupshad the ability to compare their lab experiences with CSTutor to the labs where CSTu-tor was not used. At the end of the semester we gave an exit survey consisting of threeopen-ended questions, shown in Table VI. The results of the exit survey are shown inTable VII. The results indicate that the majority of the CSTutor-projector treatmentgroup found CSTutor more effective and engaging, and it was also what the majoritypreferred. However in the CSTutor-user treatment group the majority found CSTutormore effective and engaging, although they did not prefer CSTutor to their regularlab. From the comments in the exit survey, it was clear that the majority CSTutor-user


�

�

�

�

�

�

�

�


Table VI. Experiment 2 Exit Survey Questions

The students were asked to compare the labs that used CSTutor to their standard labs.# Question1 Which lab format (with CSTutor or without) do you think was more effective at teaching data

structures and why?2 Which lab format (with CSTutor or without) do you think was more engaging and why?3 Which lab format (with CSTutor or without) do you prefer and why?

Table VII. Experiment 2 Exit Survey Results

Treatment Response Question #1 Question #2 Question #3CSTutor Projector CSTutor 24 16 25

Non 3 6 2Both 1 1 1Neither 1 5 1% CSTutor 83% 53% 83%

CSTutor User CSTutor 21 26 10Non 15 14 26Both 2 0 3Neither 0 0 0% CSTutor 54% 65% 25%

group did not prefer CSTutor due to the time constraints. Many students said in theircomments that they needed more time to go over how CSTutor worked and how to useit during the lesson.

Many users commented on the usefulness of creating code from a drawing, such asthe following.

“Yes, takes you through the process easily and writes the code for you bydrawing.”

“Yes, drawing could give code so that students can better understand how towrite code based on theory knowledge.”

In addition, many students commented on the limitations of the sketching, such as notbeing able to draw a full program.

“Only in certain situations. Students can’t draw programs out using thistool.”

7. DISCUSSION

7.1. Results

The results from the first experiment support hypotheses H2 and H3. The students inthe CSTutor group had significantly better Final Exam results than the control group,but the Exam 2 results were not significantly different. These results indicate thatthe CSTutor group had better retention of the topics covered over the entire courseand partially supports the hypothesis that the CSTutor students would perform betteron course exams (hypothesis H2). However, there was no support for hypothesis H1,since there were no significant differences in quiz scores between groups. In addition,students did rate CSTutor higher for certain topics (Linked Lists Intro, Linked ListsAdvanced, Sorting, and AVL Trees), supporting hypothesis H3.

The results from the second experiment support hypotheses H6 and H7 about stu-dent preference and engagement, however there was only partial support for H4 andH5 about quiz and exam performance. The final exit survey allowed the Treatment I


�

�

�

�

�

�

�

�


and II students to compare their labs with CSTutor to the ones without CSTutor. Thisrevealed that 83% of students surveyed in the Treatment I group found CSTutor moreeffective and preferred it to their standard labs. The Treatment II group did not preferCSTutor to their standard lab (probably due to the time restrictions), however 65%thought it was more engaging than their standard labs. There was partial supportfor hypothesis H5 in that the Treatment I group had better Exam 2 results than thecontrol group, but there was not a significant difference for the Treatment II group.In addition there wasn’t a significant difference between groups on the Final Examresults, which may be because there were many more topics on the Final Exam thancovered in the second experiment. Overall the results are positive for the instructorusing CSTutor in the classroom and indicating higher ratings for certain topics. Whilestudents didn’t prefer using CSTutor in the classroom themselves, this was mainly dueto the time constraints involved and we believe this method has potential given moretime.

The results from the second experiment were different than the first in the examscores. We postulate that in the first experiment the treatment group had significantlyhigher final exam scores than the control group, but not the second experiment, sincethe first experiment treatment group had longer exposure to CSTutor (7 weeks versus4 weeks).

The second experiment expanded upon the first experiment by focusing on severalissues we wanted to explore. First, did using CSTutor on the projector aid in the pre-sentation of material? Second, did the pen interface aid the instructor or the studentsby providing an intuitive interface? Third, would CSTutor be better suited as a lectureaid or a student tool?

7.1.1. Presentation of Material. Many students commented that they liked the visualiza-tions of the material and that it helped them better understand the data structures,in Sections 5.5.2 and 6.5.2. Other observations were that the instructor could easilycreate examples on the fly when students wanted to see different examples in actionor asked “What if we did this instead?” The instructor could then easily sketch anoperation on the diagram and show the code created, or could change the code anddemonstrate the changes on the diagram.

7.1.2. Pen Interface. Many users commented on the usefulness of creating code fromdrawing, such as in Section 6.5.2. Of the CSTutor user group 65% rated CSTutor moreengaging than a normal whiteboard lab, providing evidence that students enjoy thetype of pen-based interaction provided by CSTutor. The instructor also had the benefitof being able to create animations and annotate at any time during the animationusing the pen interface. However, recognition errors (such as inserting a 17 into aBST instead of 11) or robustness issues occurred but could be improved with more finetuning.

7.1.3. Lecture Aid versus Student Tool. The CSTutor user group rated CSTutor 65% moreengaging than a normal whiteboard lab, with 53% of the projector group, indicatingthat CSTutor could be successful as a student tool or lecture aid. In addition, 83% ofthe projector group found CSTutor more effective than their regular lab, as did 54% ofthe user group. However, only 25% of the user group preferred CSTutor to their regularlab, versus 83% of the projector group. Although these results indicate that CSTutoris effective as a lecture aid, the results are mixed for using CSTutor as a student tool.Since many of the comments also indicated users would have to spend more time withthe tool to get comfortable enough to use it in a regular lab setting, we postulate thatif students spend more time with the tool it would be an effective student tool.


�

�

�

�

�

�

�

�


7.2. CSTutor Limitations

Currently the user cannot generate a routine or function from sketching since theCSTutor does not deduce an algorithm from the drawing. The user can primarily in-teract with the data structure in two ways. First, the user can sketch basic operations,such as adding, deleting, or rearranging nodes, that will generate basic code steps.Second, the user can sketch a data structure and then type in code to manipulate thatdata structure. The user can then step through each line of code as it manipulates thatdata structure. By using method (2), the user can test and visualize any algorithmsthat manipulate the chosen data structure.

Since CSTutor only supports the C language it is not available to the widest audi-ence. Any C code can be used, although the diagram interpreter assumes that the userwill follow each data structure’s paradigm (a BST and AVL tree have a left and rightchild, while a Linked List has a next node). Additionally, there is a learning curve as-sociated with learning the interaction modes with each data structure that will takethe user some time to get used to.

CSTutor requires a tablet PC running Windows 7 with a stylus and keyboard. Thus,CSTutor does not leverage the growing availability of multi-touch tablets. However,it does serve as a proof-of-concept system that could be eventually rewritten in Javaor Javascript to allow access by any touch or stylus-enabled tablet or PC. Since manybrowsers are now equipped with touch events, CSTutor would make a good Web ap-plication that could provide mobile access in addition to saving and loading data. Theissue is that the symbol (numbers) and gesture (circles, rectangles, lines, and arrows)recognition provided by a stylus is more accurate than touch input. However, sinceCSTutor uses a small set of symbols (numbers only) and gestures, replacing stylus in-put with finger input does not seem to be an issue. In order to explore the feasibilityof other platforms, we have tried using the touch input provided by some of our newertablets and have seen only slightly worse recognition results. In addition, MacLeanet al. [2011] found in their experimental comparisons that multi-touch devices are aviable platform for sketch-based interfaces. This leads us to the conclusion that CSTu-tor has the potential to be a widely used tool if it is implemented in a cross-platformlanguage that leverages popular multi-touch technology.

7.3. Future Work

We are encouraged by the results we have achieved thus far, but would like to makeCSTutor a more widely available and all encompassing tool for students. In order tomake it widely available, we would like to make a companion Web version as well asa touch enabled version for mobile tablets. In addition, we would like to incorporateall additional data structures and algorithms seen in CS1 and CS2 courses as well asdifferent programming languages so that CSTutor can be used outside of CS1 at UCF.We would also like to make the generated code more modular and to add the ability todraw out algorithms, allowing students to create code for algorithms by sketching outincremental operations. It would also be interesting to evaluate the specific benefits ofa sketching interface as opposed to a mouse and keyboard interface.

8. CONCLUSION

The integration of a sketch-based interface, coding area, and data structure visual-izations in CSTutor is intended to enhance teaching and learning experiences in datastructures courses. CSTutor was evaluated in two semester long studies and was anexciting addition to student’s lab classes. Many students indicated that it helped theiroverall comprehension of CS1 topics, they preferred it, and were more engaged using


�

�

�

�

�

�

�

�


it than having a standard whiteboard lecture. After proving that CSTutor is a usefultool for instructor use in the classroom in the first experiment, we wanted to show thatCSTutor was a useful tool for student use in the classroom in the second experiment.However, our results indicated that students would need more exposure to CSTutor togain the proper benefits (instead of only 30 minutes during lab). Future work will fo-cus on making CSTutor more widely available to students by extending it to differentplatforms, as well as adding different programming languages and data structures. Webelieve this work is a good foundation for exploring this type of pen-based approach foralgorithm visualization and will encourage more widespread use.

APPENDIX A.

Advanced Linked List Pretest

For the following question use the following struct for a node in a linked list:

struct node {int data;struct node* next;

};

1) p is a pointer that points to the first element in a linked list that contains theelements: 66, 9, 14, 52, 87, 14 and 17 in that order. Consider running the following lineof code:

p = mystery(p);

where mystery is the function defined below.

struct node* mystery(struct node *list) {

struct node* a = list;struct node* b = list;struct node* c;

if (a == NULL)return NULL;

while ( a->next != NULL)a = a ->next;

a->next = b;c = b->next;b->next = NULL;

return c;}

a) Show the contents of p after the function call.

b) Describe in words what the function is doing.


�

�

�

�

�

�

�

�


Advanced Linked List Quiz (10 pts)

For the following questions use the following struct for a node in a linked list:

struct node {int data;struct node* next;

};

1) (5 pts) p is a pointer that points to the first element in a linked list that con-tains the elements: 77, 10, 12, 48, 98, 15 and 19, in that order. Consider running thefollowing line of code:

p = mystery(p);

where mystery is the function defined below.

struct node* mystery(struct node *list) {struct node* a = list;struct node* b = list;struct node* c;

if (a == NULL) return NULL;

while ( a->next->next != NULL)a = a ->next;

c = a->next;a->next = NULL;c->next = b;

return c;}

a) Show the contents of p after the function call.

b) Describe in words what the function is doing.

2) (5pts) Write a function that deletes the node that comes after every node that hasa value of 6 in a linked list, and returns a pointer to the front of the list. If there areno items in the original list, NULL should be returned.

struct node* deleteAfter6(struct node* list) {

}

For the following questions:

— Respond to each question with whatever knowledge you have.— There are no right or wrong answers.— Be honest and realistic in your assessment.


�

�

�

�

�

�

�

�


— State your agreement based on the following scale:

1 2 3 4 5 6 7Strongly Disagree Strongly Agree

Table VIII. Example Survey Questions Given to the Treatment Groups

Perceived Usefulness:1) CSTutor is helpful in understanding the implementation of Linked Lists. 1 2 3 4 5 6 72) CSTutor ihelps in visualizing the steps in Linked List functions. 1 2 3 4 5 6 73) CSTutor makes understanding Linked Lists difficult. 1 2 3 4 5 6 74) Other Comments about CSTutor:

Table IX. Example Survey Questions Given to the Control Group

Perceived Usefulness:1) The lecture was helpful in understanding the implementation of Linked Lists. 1 2 3 4 5 6 72) The lecture helped in visualizing the steps in Linked List functions. 1 2 3 4 5 6 73) The lecture made understanding Linked Lists difficult. 1 2 3 4 5 6 74) Other Comments about the lecture:

REFERENCES

Ayonike Akingbade, Thomas Finley, Diana Jackson, Pretesh Patel, and Susan H. Rodger. 2003. JAWAA:Easy web-based animation from CS 0 to advanced CS courses. SIGCSE Bulletin 35, 1, 162–166.

Lisa Anthony and Jacob O. Wobbrock. 2010. A lightweight multistroke recognizer for user interface proto-types. In Proceedings of Graphics Interface (GI). 245–252.

Jens Bennedsen and Carsten Schulte. 2010. BlueJ visual debugger for learning the execution of object-oriented programs. Trans. Comput. Educ. 10, 2.

M. H. Brown and R. Sedgewick. 1985. Techniques for algorithm animation. IEEE Softw. 2, 1, 28–39.Sarah Buchanan, Brandon Ochs, and Joseph J. LaViola Jr. 2012. CSTutor: A pen-based tutor for data struc-

ture visualization. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Educa-tion (SIGCSE). 565–570.

Michael D Byrne, Richard Catrambone, and John T Stasko. 1999. Evaluating animations as student aids inlearning computer algorithms. Comput. Educ. 33, 4, 253–278.

Travis J. Cossairt and Joseph J. LaViola, Jr. 2012. SetPad: A sketch-based tool for exploring discrete mathset problems. In Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling(SBIM). 47–56.

Pilu Crescenzi and Carlo Nocentini. 2007. Fully integrating algorithm visualization into a cs2 course: A two-year experience. In Proceedings of the 12th Annual SIGCSE Conference on Innovation and Technologyin Computer Science Education (ITiCSE). 296–300.

Martha E. Crosby and Jan Stelovsky. 1995. From multimedia instruction to multimedia evaluation. J. Educ.Multimedia Hypermedia 4, 2, 147–62.

James H. Cross, II, T. Dean Hendrix, Jhilmil Jain, and Larry A. Barowski. 2007. Dynamic object viewersfor data structures. In Proceedings of the 38th SIGCSE Technical Symposium on Computer ScienceEducation (SIGCSE).

James H. Cross, II, T. Dean Hendrix, David A. Umphress, Larry A. Barowski, Jhilmil Jain, and Lacey N.Montgomery. 2009. Robust generation of dynamic data structure visualizations with multiple interac-tion approaches. Trans. Comput. Educ. 9, 2.

David Furcy. 2009. JHAVEPOP: Visualizing linked-list operations in C++ and Java. J. Comput. Sci. Colleges25, 32–41.

Scott Grissom, Myles F. McNally, and Tom Naps. 2003. Algorithm visualization in CS education: Compar-ing levels of student engagement. In Proceedings of the ACM Symposium on Software Visualization(SoftVis).

Tracy Hammond and Randall Davis. 2006. Tahuti: A geometrical sketch recognition system for UML classdiagrams. In Proceedings of the ACM SIGGRAPH Courses.


�

�

�

�

�

�

�

�


Christopher D. Hundhausen and Jonathan L. Brown. 2007. What you see is what you code: A live algorithmdevelopment and visualization environment for novice learners. J. Vis. Lang. Comput. 18, 1, 22–47.

Christopher D. Hundhausen, Sarah A. Douglas, and John T. Stasko. 2002. A meta-study of algorithm visu-alization effectiveness. J. Vis. Lang. Comput. 13, 3, 259–290.

Bo Kang and Joseph LaViola. 2012. LogicPad: A pen-based application for visualization and verificationof Boolean algebra. In Proceedings of the ACM International Conference on Intelligent User Interfaces(IUI).

Charles Kann, Robert W. Lindeman, and Rachelle Heller. 1997. Integrating algorithm animation into alearning environment. Comput. Educ. 28, 4, 223–228.

Ville Karavirta, Ari Korhonen, Lauri Malmi, and Kimmo Stalnacke. 2004. MatrixPro-A tool for on-the-fly demonstration of data structures and algorithms. In Proceedings of the 3rd Program VisualizationWorkshop. 26–33.

Colleen Kehoe, John Stasko, and Ashley Taylor. 2001. Rethinking the evaluation of algorithm animations aslearning aids: An observational study. Int. J. Hum.-Comput. Studies 54, 2, 265–284.

A. W. Lawrence, A. M. Badre, and J. T. Stasko. 1994. Empirically evaluating the use of animations to teachalgorithms. In Proceedings of the IEEE Symposium on Visual Languages. 48–54.

Andrea W. Lawrence. 1993. Empirical studies of the value of algorithm animation in algorithm understand-ing. Tech. rep., DTIC Document.

S. MacLean, D. Tausky, G. Labahn, E. Lank, and M. Marzouk. 2011. Is the iPad useful for sketch input?A comparison with the tablet PC. In Proceedings of the 8th Eurographics Symposium on Sketch-BasedInterfaces and Modeling (SBIM).

Andres Moreno, Niko Myller, Erkki Sutinen, and Mordechai Ben-Ari. 2004. Visualizing programs withJeliot 3. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI). 373–376.

T. L. Naps, J. R. Eagan, and L. L. Norton. 2000. JHAVE—An environment to actively engage students inWeb-based algorithm visualizations. ACM SIGCSE Bull. 32. 109–113.

Thomas L. Naps, Guido Rossling, Vicki Almstrum, Wanda Dann, Rudolf Fleischer, Chris Hundhausen, AriKorhonen, Lauri Malmi, Myles McNally, Susan Rodger, and J. Angel Velazquez-Iturbide. 2002. Explor-ing the role of visualization and engagement in computer science education. Working Group Reportsfrom ITiCSE on Innovation and Technology in Computer Science Education (ITiCSE-WGR).

Jussi Nikander, Ari Korhonen, Otto Seppala, Ville Karavirta, Panu Silvasti, and Lauri Malmi. 2004. Visualalgorithm simulation exercise system with automatic assessment: TRAKLA2. Inform. Educ. Intern. J.3, 2, 267–288.

Michael C. Orsega, Bradley T. Vander Zanden, and Christopher H. Skinner. 2012. Experiments with algo-rithm visualization tool development. In Proceedings of the 43rd ACM Technical Symposium on Com-puter Science Education (SIGCSE).

S. L. Oviatt and A. O. Cohen. 2010. Toward high-performance communications interfaces for science problemsolving. J. Sci. Educ. Technol. 19, 6, 515–531.

S. Oviatt, A. Cohen, A. Miller, K. Hodge, and A. Mann. 2012. The impact of interface affordances on humanideation, problem solving, and inferential reasoning. ACM Trans. Comput.-Hum. Interact. 19, 3.

Roy P. Pargas and Samuel Bryfczynski. 2009. Using ink to expose students’ thought processes in CS2/CS7.SIGCSE Bull. 41, 1, 168–172.

L. Qiu. 2007. SketchUML: The design of a sketch-based tool for UML class diagrams. In Proceedings ofWorld Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA).

G. Roßling and B. Freisleben. 2001. Program visualization using AnimalScript. In Proceedings of the 1stInternational Program Visualization Workshop. 41–52.

Thomas A. Standish. 1995. Data Structures, Algorithms, and Software Principles in C. Addison-Wesley,Reading, MA.

John Stasko. 1990. Tango: A framework and system for algorithm animation. Comput. 23, 9, 27–39.John Stasko. 1992. Animating algorithms with XTANGO. SIGACT News 23, 2, 67–71.John Stasko, Albert Badre, and Clayton Lewis. 1993. Do algorithm animations assist learning? An empir-

ical study and analysis. In Proceedings of the INTERACT and CHI Conference on Human Factors inComputing Systems (CHI). 61–66.

Jaime Urquiza-Fuentes and J. Angel Velazquez-Iturbide. 2012. Comparing the effectiveness of different ed-ucational uses of program animations. In Proceedings of the 17th ACM Annual Conference on Innovationand Technology in Computer Science Education (ITiCSE). 174–179.

J. Angel Velazquez-Iturbide. 2012. Refinement of an experimental approach tocomputer-based, active learn-ing of greedy algorithms. In Proceedings of the 17th ACM Annual Conference on Innovation and Tech-nology in Computer Science Education (ITiCSE). 46–51.


�

�

�

�

�

�

�

�


Yiyan Xiong and Joseph J. LaViola, Jr. 2009. Revisiting ShortStraw: Improving corner finding in sketch-based interfaces. In Proceedings of the 6th Eurographics Symposium on Sketch-Based Interfaces andModeling (SBIM). 101–108.

Zhenming Yuan, Hong Pan, and Liang Zhang. 2008. A novel pen-based flowchart recognition system forprogramming teaching. In Advances in Blended Learning, E. Leung, F. Wang, L. Miao, J. Zhao, andJ. He Eds., Lecture Notes in Computer Science, vol. 5328, Springer Berlin, 55–64.

Received October 2012; revised June 2013, September 2013; accepted October 2013


Documents

CSTutor: A Sketch-Based Tool for Visualizing Data Structures · the user edits the code, the diagram will animate and update itself. The data structures represented in CSTutor (Linked