AC 2010-2210: EVALUATION OF RAPID DEVELOPMENT SYSTEM USING EYETRACKER

Arun Chintalapati, Missouri University of Science and Technology

Hong Sheng, Missouri University of Science and Technology

Richard Hall, Missouri University of Science and Technology

Robert Landers, Missouri University of Science and Technology

© American Society for Engineering Education, 2010

Page 15.541.1

Evaluation of Rapid Development System using Eye Tracker Abstract This paper presents the results of the evaluation of Linear Axis Rapid Development System10 (RDS), which is under development as part of a NSF funded project. The Linear Axis RDS is used in teaching control design/insertion in the Mechanical Engineering curriculum at a mid-sized midwestern university in the United States. The Linear Axis RDS has a graphical user interface with three main modes: simulate, emulate, and implement. The objective of this evaluation was to test the overall effectiveness of the Linear Axis RDS. A combination of qualitative and quantitative research methods were applied in the evaluation of thirty-four participants from the Mechanical and Aerospace Engineering Department at Missouri University of Science and Tech. The Technology Acceptance Model, a model that has been used extensively to study acceptance of technology was used to guide the study. Learning Styles and Learning Outcomes were added to study the learning effects of the system. Eye tracking was used in two of the tasks to provide both qualitative and quantitative data. Eye tracking is an innovative method that is increasingly being used in the field of human-computer interaction for usability studies, as it can provide useful insight into the cognitive aspect of the users. Based on the data analysis, a significant improvement was noticed in users interest after using RDS. Statistical analysis showed significant increase in career interest in science followed closely by enjoyment. Results from the analysis on learning outcomes suggest the RDS was perceived to have high real world applicability. Results also showed an increase in knowledge gained after using the system. The Technology Acceptance Model (TAM) constructs such as perceived ease of use (PEOU), perceived usefulness (PU), attitude (ATT) and intention to use (INT) were found to influence the learning outcome. Eye tracking results validated the results from the survey analysis. The gaze plots and heat maps indicated that the participants were able to identify important areas of the interface, such as tip box and help button, which were newly developed. Overall the results suggest that RDS is well received by the participants and is an effective learning tool I. Introduction Engineering education has benefitted tremendously from the use of information technology. One of the key aspects of engineering education is hands-on experiments, either using the main equipments or by using tools that simulate the equipments.2The use of IT tools to simulate the equipment enhances the quality of education as well as accommodates students with various learning styles.2

An important aspect of such educational software is that it should be easy to use, learn and understand. In essence it should be usable. Usability is defined as “the capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions.”1 Usability testing plays an important role in the development of interactive educational software and user-centered design is quintessential for promoting its usage. “Usability problems of educational software can be one source of disturbance within the learning process by distracting attention from the learning task and consequently increasing the extraneous cognitive load” 4

Page 15.541.2

In this study, a comprehensive usability evaluation was performed on a computer based learning software for engineering students. The objective of the evaluation was to test if users would accept the RDS system as a learning tool. Combinations of qualitative and quantitative research methods were applied in the evaluation, including survey, eye tracking, and think-aloud protocol. III. Research Model:

The research model15 was developed from previous models incorporating various constructs from different literature. The proposed model can be used in evaluating IT based learning technologies, user’s acceptance of the technology and the learning outcome associated with the usage of the technology. One of the models that have been extensively used in prior literature for studying the user acceptance of technology is the technology acceptance model (TAM). 3 The Technology acceptance model 3 suggests that perceived ease of use of the technology together with perceived usefulness influences user’s attitude towards the technology and consequently the intention to use it. Perceived usefulness is defined as "the degree to which a person believes that using a particular system would enhance his or her job performance” 3 and perceived ease of use of the technology is "the degree to which a person believes that using a particular system would be free of effort." Perceived ease of use and perceived usefulness of the technology would determine the user’s attitude towards the technology, which in turn would affect the users’ intention to accept the technology. Attitude is defined as “the favorable or unfavorable feeling towards that particular behavior” 8.

Page 15.541.3

As TAM has proven to be an effective model in predicting users’ intention to adopt or use new technology, it can be applied to study the acceptance of the RDS, which is a relatively new technology. This model was extended by including new variables learning outcome and learning style.

Learning Styles

Learning styles refer to the various ways students prefer to receive information as well as how they process the information. Engineering education should incorporate instructions that accommodate students with different learning styles 4. Previous research identified the usage of information technology as a way to address the dissemination of knowledge to accommodate students with various learning styles.1

Felder and Silverman 6 categorized learning styles as follows:

1. The type of information student prefers to perceive: sensory (facts, practical training) or intuitive (abstractions-theories, models).

2. The type of information that can be most effectively perceived: visual (pictures, diagrams etc) or verbal (written or spoken explanation)

3. Their preference in processing the information: actively (engagement in discussion or physical activity) or reflectively (introspection)

4. Their progress towards understanding: sequential (step by step incrementally) or globally (large jumps)

Learning outcomes

Learning outcome refers to the perceived knowledge gained by the students through the use of different methods of teaching or studying. Methods of teaching or studying typically include reading of textbooks, performing experiments in laboratory setting, and using information technologies. Assessment of learning outcomes often include quantitative measures for notions such as motivation to learn, real world applicability, and knowledge or learning awareness 13. IV. Research Techniques Eye tracking and its role in usability evaluations: Eye tracking is a technique in which an individual’s eye movements are measured in order to understand where an individual is looking as well as the sequence in which he/she shifts from one location to another location. In the usability field, these measures would help researchers ascertain various factors that would impact the usability of the system 15.

One of the primary reasons why eye tracking is used in usability studies is because of its relationship with the cognitive process. The “eye-mind” hypothesis 15 indicates that what a person observes is assumed to indicate the thought “on top of the stack” of cognitive process 14. This means that eye movement recordings can provide researchers with a trace of the user’s

Page 15.541.4

attention on an interface. Other eye tracking measures such as fixations (stationary eye movements) can indicate the processing time on the area that was fixated 14. In combination with other usability measures such as speed of task, user behavior (observation,) eye tracking can provide useful insight into the users’ cognition10

The eye tracking data can be analyzed using gaze plots, heat maps, and areas of interest. From a qualitative perspective, eye tracking fixation data can be studied using gaze plots and heat maps. Gaze plots provide static view of the gaze points, which are fixation points that are numbered in sequence. The gaze points are colored in blue with the radius indicating the length of the fixation. These plots help visualize scan paths, which can be defined as “spatial arrangement of a sequence of fixation” 12 as well as saccades which can be defined as “An eye movement occurring between fixations, typically lasting for 20 to 35 milliseconds”16. Heat maps or hot spot graphs indicate the user’s focus using the temperature analogy by using darker colors to represent intense focus and lighter color to represent less focus. To obtain specific quantitative data researchers create areas of interest, which can be defined as “area of display or visual environment that is of interest to the research or design team and thus defined by them (not by the participant)” 12. These are defined over specific areas of the interface. Researchers quantify the eye movements in this area that are objectively evaluated for visibility, meaningfulness and the placement. Thus, providing researchers with findings that can be used in evaluating the design of the interface .9

Thus, eye tracking can be used in examining how users adapt to unfamiliar layouts as well as where and how they look for information. V. Other Research Techniques used in the evaluation Triangulation is an important aspect of usability testing as it helps get inputs from multiple methods as well as converge the problem areas associated with the interface. Studies indicate that using triangulation has helped find more usability problems. 4, 16

In additional to eye tracking, a few other research techniques were also used in the evaluation.

• Think Aloud Protocol: Think aloud protocols involve participants thinking aloud as they are performing a set of specified tasks. Users are asked to articulate whatever they are looking at, thinking, doing, and feeling, as they go about performing the task.

• Questionnaire: A pre/post questionnaire (Appendix 1,2) was used as a part of the evaluation. The pre questionnaire had questions that addressed various constructs such as Learning Styles 5 questions on Interest in Science. The post questionnaire had questions from the Technology Acceptance Model 2, Learning Outcomes and Interest in Science. All the constructs except Learning Style had 7-point likert scale rating.

VI. Project Background Linear Axis Rapid Development Phase-II (fig1a, b, c):

Page 15.541.5

The Linear Axis RDS is used in teaching materials on control design/insertion in the Mechanical Engineering curriculum at a midwestern university in the U.S. It has a graphical user interface with three main modes: simulate, emulate, and implement. In the simulation mode the student simulates the linear axis system that includes their controller and detailed models of the interface hardware and linear axis. In the emulation mode, the simulation is performed on the computer hardware that will implement the controller. In the implementation mode, the controller is deployed on the hardware system and experimental data is gathered10. The physical linear axis position can be adjusted by pushing the jog button on the main interface Figure 1a. This opens the jogger display, shown in Figure 1b, which allows the student to move the linear axis in the positive or negative direction in three incremental. The target computer that acts as an interface between the hardware Figure 1c, utilizes the xPC real–time operating system. A few improvements had been made on the interface using the results from a prior evaluation. The overall interface was made wider. An interactive help menu was created to provide step-by-step help instructions to the user. This menu was made accessible by clicking a prominently visible help button. A second inclusion was a tip box, which provided the user with simple guidelines on the usage. The third inclusion was feedback box that flashed messages, which guided the user in the initial stages. The latest inclusion was the email button at the bottom through which the student could email their data files.

1b. Jog Screen 1a. Home screen

Page 15.541.6

The participants for this research were senior students enrolled for the “ME 279: Automatic Control of Mechanical Systems” class. This is a required course under the Mechanical curriculum for undergraduate students. The course is designed to introduce senior students to the basics of designing and analyzing feedback control systems using classical control methods. A semester project exists for students to design a control system to satisfy a given set of specifications for a system MATLAB and SIMULINK software packages are used as design tools in this course. The outcomes associated with this course are as follows (a) An ability to apply knowledge of mathematics, science, and engineering, (c) An ability to design a system, component, or process to meet desired needs. (e) An ability to identify, formulate, and solve engineering problems. (h) The broad education necessary to understand the impact of engineering solutions in a global and societal context. (i)A recognition of the need for, and an ability to engage in life-long learning. (k) An ability to use the techniques, skills, and modern engineering tools necessary for

1c.  Simulate  Function  

1d.  Mini  CNC  hardware  

Page 15.541.7

engineering practice. (m) An ability to apply advanced mathematics through multivariate calculus and differential equations (n) Familiarity with statistics and linear algebra. RDS was introduced to the students as lab assignment to simulate the controller that was designed by them as part of the course requirement. VII. Research Procedure A total of 34 students participated in the complete experiment. All the participants were senior undergraduate students from the mechanical department. Out of the 34 participants, 31 were in the 18-25 age group and the remaining 3 were in the 25-38 age group. Among the participants there were 31 males and 3 females. The research used a pre-post experimental design, in which a pre questionnaire was distribute one month prior to the experiment. A post questionnaire was given to the students after they completed the experiment. Data from the students who completed both the pre and post questionnaire was used in the data analysis. One week prior to the experiment which involved the usage of RDS system students were asked to develop their controllers using the Simulink syntax, which is a visual syntax consisting of blocks and connectors that can be used to create the student’s controller to use with the RDS. In the experimental evaluation participant were shown the setup. The participants were asked to sign the consent form before they began the experiment after which they were given a brief introduction of the RDS and the research techniques involved in the evaluation. A list of 4 tasks was given to every participant. They are as follows: TASK 1: Use the Jog function. TASK 2: Simulate your controller. Generate model. TASK 3: Emulate your controller. Generate model. TASK 4: Implement your controller. The students were given 30 min each to complete all the four tasks. The tasks 2,3,4 required the students to use the controller they developed. Eye-tracker was used to record the first and the fourth tasks. The participant was asked to think-aloud when he or she was using the RDS to complete the second and the third task. When the participant completed all the four tasks, he or she was asked to complete a post questionnaire. After the participant completed the questionnaire, an interview was conducted.

Page 15.541.8

IX. Results Summary Task Performance: The table below summarizes the task performance of the students.

Table Task performance

Task Description No. Of participants who completed (out of 34)

1 Use the Jog function 33

2 Simulate and select the controller. Generate model. 34

3 Emulate and select the controller. Generate model. 34

4 Go back and implement a preexisting controller. 34

Almost all the participants were able to complete all the tasks without the need for any prompt. Some of the participants had issues with their controller. Quantitative Analysis: Quantitative data obtained from the pre and post questionnaire (Appendix 1,2) data was analyzed to check for significant results. All the constructs had 7-point likert scale questions except for the Learning Styles construct. The pre-questionnaire had a total of 40 questions in which were divided into three section. Part I collected data on the Learning Styles of the student. A total of 12 questions were used to categorize the learning styles of the students. Part II of the questionnaire contained questions on the Interest in science constructs. The questions were categorized into three areas namely general interest in science, career interest in science and enjoyment in science. Part III collected the demographic data of the students. The post-questionnaire had a total of 40 questions that collected data on various aspects of the technology acceptance model such as perceived ease of use, perceived usefulness, attitude towards the software and intention to use. The questions also collected data on the learning outcomes as well as the interest in science constructs. Students were asked to include their names in the questionnaire; this was used in comparing the data between pre and post.

Page 15.541.9

1. Learning Styles:

The above chart validates the result from previous that most engineering students are visual, sensing and active.5 Among all the 34 students who participated in the experiment, 26 of them were found to be visual learners, 20 of them were active learners, 31 out of 34 students were sensory learners and finally 31 out of the 34 students were sequential learners.

2. Interest in science:

The Analysis of interest in science constructs Pre (before being exposed to RDS) and Post (after being exposed to RDS) showed significant improvements. Career interest in science showed the maximum improvement followed by enjoyment in science.

76.47  

23.52  

58.82  

41.17  

82.35  

17.64  

91.17  

8.82  

0  10  20  30  40  50  60  70  80  90  100  

Learning Styles %

4.0735 4.035 4.64

4.264 4.5735 5.1

0

1

2

3

4

5

6

7

General interst in science

Career interest in science

Enjoyment in science

Pre Post

Page 15.541.10

Paired Samples Test

Paired Differences

95% Confidence Interval

of the Difference

Mean

Std.

Deviation

Std. Error

Mean Lower Upper t df

Sig. (2-

tailed)

Pair

1

Pre General Interest

in Science - Post

General Interest in

Science

-.19118 .74635 .12800 -.45159 .06924 -1.494 33 .145

Pair

2

Pre Career Interest in

Science- Post Career

Interest in Science

-.54412 .82451 .14140 -.83180 -.25643 -3.848 33 .001

Pair

3

Pre Enjoyment in

Science – Post

Enjoyment in Science

-.46765 .85261 .14622 -.76514 -.17016 -3.198 33 .003

Paired t-tests were conducted on the interest in science constructs for the pre and post. The results indicate a significant difference in career interest in science (t = -3.848, p = .001) and enjoyment in science (t = -3.198, p = .003). Learning Outcome:

Analysis of the results from learning outcome showed the real world applicability of the RDS system to be high in comparison with lecture and notes.

4.61  3.76  

5.7  5.53  

3.94  5.08  4.76  

3.82  

4.61  

0 1 2 3 4 5 6 7

RDS  

Lecture  

Notes  

Page 15.541.11

Multivariate Tests (b)

Value F Hypothesis df Error df Sig.

Learning Pillai's Trace 0.185 3.621a 2 32 0.038

Wilks' Lambda 0.815 3.621a 2 32 0.038

Hotelling's Trace 0.226 3.621a 2 32 0.038

Roy's Largest Root 0.226 3.621a 2 32 0.038

Motivation Pillai's Trace 0.036 .598a 2 32 0.556

Wilks' Lambda 0.964 .598a 2 32 0.556

Hotelling's Trace 0.037 .598a 2 32 0.556

Roy's Largest Root 0.037 .598a 2 32 0.556

Real World Applicability (RWE) Pillai's Trace 0.46 13.648a 2 32 0

Wilks' Lambda 0.54 13.648a 2 32 0

Hotelling's Trace 0.853 13.648a 2 32 0

Roy's Largest Root 0.853 13.648a 2 32 0

a. Exact statistic b. Design: Intercept

Results show a significant multivariate effect for real world applicability and learning. A one-way MANOVA revealed a significant multivariate main effect for real world applicability, Wilks’ λ = .54, F (2, 32) = 13.648 and learning, Wilks’ λ =0.815, F (2, 32) =3.621. Results also showed that students learned more from lectures and notes. This can be attributed to the following factors: Firstly all the participants were undergraduate students. Secondly the assignment was not part of their course requirement.

Page 15.541.12

Students also felt that they gained more knowledge after using RDS, which establishes the fact that RDS can be used as part of the course-curriculum and proves its role as a learning tool. Paired t-test was conducted on the knowledge before and after using the RDS, results showed a significant difference in the knowledge gained (t= -6.076, p = .000). Paired Samples Test

Paired Differences 95% Confidence Interval of the Difference

Mean Std. Deviation

Std. Error Mean Lower Upper t df

Sig. (2-tailed)

Pair 1

Knowledge Gained Before Using RDS – Knowledge Gained After Using RDS

-1.26471 1.21378 .20816 -1.68821 -.84120 -6.076 33 .000

Technology Acceptance Constructs (TAM):

3.1176  4.3823  

0  

1  

2  

3  

4  

5  

6  

7  

Knowledge  gained-­‐Before   Knowledge  gained-­‐AJer  

5.1274 4.8455 4.941

2.9705

0 1 2 3 4 5 6 7

Perceived  Ease  of  Use  

Perceived  Usefulness  

AOtude   IntenQon  to  use  

Page 15.541.13

The results reported a relatively low intention to use RDS. One of the plausible reasons, according to the results from the interviews, was that can be attributed to the fact that all the participants were undergraduate students and most of them didn’t expect to use RDS in classes other than ME 279. Eye Tracking Results Eye tracking results validated the quantitative data and qualitative data. Fig 4a shows that the user immediately noticed the Jog button as part of the first task in which the user had to use the jogger function. Fig 4b shows that the user noticed all his selections but skipped the feedback assuming his controller had been loaded. Unfortunately most of the participants assumed that their controller was loaded without checking for feedback.

Fig 4. a Fig 4.b

Analysis of heat map from one of the participants suggested that all the areas of the interface received attention. Fig 5 a shows the heat map of the jog function. Fig 5b shows that the help menu received good attention.

Fig 5.a Fig 5.b

Figure 6a shows that the user did not read the pop-up message that has important data regarding the saved data file. This was observed in multiple participants, which was a cause of concern.

Page 15.541.14

Fig 6.a Fig 6.b

Figure 6b shows that the user did not read the pop-up dialogue box that suggested the user to verify whether he/she checked the help page before inserting the controller. Most of the participants did not read the dialogue; this can be observed in the gaze plot in 6b.

Fig 6.c

Figure 6c shows the gaze plot of a participant on encountering an error message, which pops up when xpc is not connected to the target cnc system. The gaze plot indicates that frustration of the user, which can be noticed from the rapid saccades (Appendix 3). X. Recommendations: Based on the evaluation results, we come up with the following recommendations to improve the interface of the Linear Axis RDS.

1. Remove or change the pop-up box that appears when a student is attempting to insert his

controller. Eye tracking data (Fig 6b) indicates that the user does not read the message that pops when he/she is inserting the controller. Eye tracking data from multiple participants revealed similar pattern.

2. The development team should make sure the error “connecting xpc to target”(interfacing

problem between Mini CNC hardware and computer running XPC operating system) does not pop up when the user is performing tasks as it can cause the user to be frustrated and fearful. User should be provided a button to directly connect the xpc to target or user should be transferred directly to the help page that provides instructions on how to reconnect the

Page 15.541.15

system once the user clicks on the dialogue button. Eye tracking data (Fig 6c) indicates that the user was frustrated; this can be observed from the images, where user rapid saccade movement is observed.

3. The dialogue box after saving should be made bigger to demand attention. Or data file should

be included with the legend. Eye tracking data (Fig 6a) indicates the user does not read the dialogue from the pop-up box. This was observed in the eye tracking data of multiple participants

XI. Results Discussion: We believe that the increase in interest was noticed because student felt that they gained practical exposure to the subject after using the system. According to the instructor of this course, the subject area is very theoretical and abstract; therefore, any activity that would help the students gain practical exposure in the highly theoretical subject would improve their interest. We acknowledge that a larger sample is always more desirable. Fortunately we had over 20 students in each group, which is sufficient in testing for statistical significance XII. Conclusions: This paper presents the use of eye tracking in evaluation of a learning technology namely, RDS. Eye tracking and other research techniques were used in a large-scale test to evaluate the software. The test validated the use of eye tracking in understanding users cognitive aspects and helped in providing suggestions that would help in further refinement and improvement of the interface. This would enhance the acceptance of the technology and improve the learning outcomes of students. Further analysis of qualitative data from the interview and think aloud protocol would f add to the results of the analysis. Data from the Control group would help in validating the results that would bolster the use of RDS as a learning tool in teaching control/design insertion.

Page 15.541.16

XII. References:

1. Bevan, N. "International Standards for Hci and Usability," International Journal of Human Computer Studies), 2001.

2. Carlson, L., and Sullivan, J. "The Integrated Teaching and Learning Program: A Pioneering Learning

Environment for d1st Century Engineering Education", in Proceedings: Realizing the New Paradigm for Engineering Education, 1998.

3. Davis, F. "Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information

Technology," MIS Quarterly), 1989, pp. 323-340

4. Domagk, S., Hessel, S., & Niegemann, H. M. (2004). How do you get the information you need? Triangulation in usability testing: Two explorative studies. In S. Banks, P. Goodyear, V. Hodgson, C. Jones, V. Lally, D. McConnell & C. Steeples (Eds.), Networked Learning 2004. Proceedings of the Fourth International Conference (pp. 749-750). Lancaster: Lancaster University.

5. Felder, R.M., and Brent, R. "Understanding Student Differences," Journal of Engineering Education

(94:1), 2005, pp. 57-72.

6. Felder, R.M., and Silverman, L.K., “Learning and Teaching Styles in Engineering Education,” Engineering Education, Vol. 78, No. 7,1988, pp. 674–681

7. Felder, R.M., and Spurlin, J. "Applications, Reliability, and Validity of the Index of Learning Styles,"

International Journal of Engineering Education (21:1), 2005, pp. 103-112.

8. Felder, R.M., “Matters of Style,” ASEE Prism, Vol. 6, No. 4, 1996, pp. 18–23.

9. Fishbein, M., and Ajzen, I. Belief, Attitude Intention, and Behavior: An Introduction to Theory and

Research, Addison-Wesley, Reading, MA, 1975.

10. Fleming, M., Jain, V., Landers, R.G., Sheng, H., and Hall, R., 2009, “Implementation and Evaluation of a Linear Axis Rapid Development System,” ASEE Annual Conference and Exhibition, Austin, Texas, June 14–17.

11. Goldberg, H. J., & Kotval, X. P. (1999). Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics, 24, 631-645.

12. Goldberg, H. J., & Wichansky, A. M. (2003). Eye tracking in usability evaluation: A practitioner’s guide. In J. Hyönä, R. Radach, & H. Deubel (Eds.), The mind's eye: Cognitive and applied aspects of eye movement research (pp. 493-516). Amsterdam: Elsevier.

13. Hall, R.H., Philpot, T., Hubing, N. (2006) “Comprehensive Assessment of a Software Development Project for Engineering Instruction”, Journal of Learning, Technology, and Assessment, 15(5), 4 – 42.

14. Jacob, R.J.K., and Karn, K.S., (2003) “Eye tracking in human computer interaction and usability research: Ready to deliver the promises”, In The Mind's Eye: Cognitive and Applied Aspects of Eye Movement Research, J. Hyona, Radach, R., and Deubel, H., Editor, Elsevier Science. Amsterdam. p. 573-605.

15. Jain, V., Sheng, H., Hall, R.H., & Hilgers, M. (2009) "Developing an Assessment Model for Evaluating

Software Tools in Education", In Proceedings of the Fifteenth Americas Conferences on Information Systems (AMCIS), San Francisco, CA, 2009

Page 15.541.17

16. Poole A, Ball LJ. Eye tracking in human-computer interaction and usability research: current status and future prospects. In: Ghaoui C, ed. Encyclopedia of Human-Computer Interaction. Pennsylvania, PA: Idea Group Inc, 2006

17. Wilson, C. E. 2006. Triangulation: the explicit use of multiple methods, measures, and approaches for

determining core issues in product development. Interactions 13, 6 (Nov. 2006), 46-ff. DOI= http://doi.acm.org/10.1145/1167948.1167980

Page 15.541.18

Appendix 1: Pre Questionnaire: Please circle the answer that applies best to you. Part I 1. When I have to work on a group project, I first want to (a) have "group brainstorming" where everyone contributes ideas. (b) brainstorm individually and then come together as a group to compare ideas. 2. In reading nonfiction, I prefer (a) something that teaches me new facts or tells me how to do something. (b) something that gives me new ideas to think about. 3. When I get directions to a new place, I prefer (a) a map. (b) written instructions. 4. When writing a paper, I am more likely to (a) work on (think about or write) the beginning of the paper and progress forward. (b) work on (think about or write) different parts of the paper and then order them. 5. When I am learning something new, it helps me to (a) talk about it. (b) think about it. 6. I am more likely to be considered (a) careful about the details of my work. (b) creative about how to do my work. 7. When I see a diagram or sketch in class, I am most likely to remember (a) the picture. (b) what the instructor said about it. 8. It is more important to me that an instructor (a) lay out the material in clear sequential steps. (b) give me an overall picture and relate the material to other subjects. 9. In a study group working on difficult material, I am more likely to (a) jump in and contribute ideas. (b) sit back and listen. 10. I prefer courses that emphasize (a) concrete material (facts, data). (b) abstract material (concepts, theories). 11. When I meet people at a party, I am more likely to remember (a) what they looked like. (b) what they said about themselves. 12. When I solve math problems (a) I usually work my way to the solutions one step at a time. (b) I often just see the solutions but then have to struggle to figure out the steps to get to them.

Page 15.541.19

Part II For the following questions, Please indicate the extent to which you agree or disagree with the following statements. 13. I would like to belong to a science club. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 14. I would like to be a scientist. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 15. Science lessons are fun. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 16. I get bored watching science programs on TV. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 17. I would like to work with people who make discoveries in science. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 18. I like science lessons. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

Page 15.541.20

19. I would like to be given a science book or a piece of scientific equipment as a present. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 20. I would like a job in a science laboratory. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 21. School should have more science lessons each week. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 22. I like reading books about science in my leisure time. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 23. Working in a science laboratory would be an interesting way to earn a living. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 24. Science is one of the most interesting school subjects. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

Page 15.541.21

Part III In this part of the survey, we are interested in your demographic information, which we will not share. Please respond by circling your answers and filling in the appropriate blanks. 1. Your name (this is only for the purpose of data analysis and will not be shared with your instructor): _____________________________________________ 2. Your gender: A. Female B. Male 3. Your age: A. 18-25 B. 25-38 C. 39-45 D. >45 4. Please indicate your ethnicity: A. White B. Black C. Hispanic D. Asian E. Native American F. Other: ___________________________ 5. What is your major? _____________________________________ 6. Which year are you in your program? A. Freshman B. Sophomore C. Junior D. Senior E. Graduate

Page 15.541.22

Appendix 2: Post Questionnaire

Name: ________________________

Part I

For the following questions, please indicate the extent to which you agree or disagree with the following statements. 1. Using Rapid Development System enables me to accomplish tasks more quickly. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 2. Learning to operate Rapid Development System is easy for me.

1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 3. I find it easy to get Rapid Development System to do what I want it to do. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 4. Using the Rapid Development System is a good idea 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 5. Using Rapid Development System improves my learning outcome. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 6. My interaction with Rapid Development System is clear and understandable. 1 2 3 4 5 6 7 Strongly Disagree Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree nor Disagree Agree Agree

Page 15.541.23

7. I predict to use the Rapid Development System in next six month. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 8. Using Rapid development system would make it easier to understand control design/insertion.

1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

9. I find Rapid Development System to be flexible to interact with.

1 2 3 4 5 6 7

Strongly Disagree Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree nor Disagree Agree Agree 10. I like the idea of using the Rapid Development System 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

11. I find Rapid Development System useful in learning control design/insertion/insertion. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 12. It is easy for me to become skillful at using Rapid Development System.

1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

Page 15.541.24

13. I intend to use the Rapid Development System in next six month. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

14. I find Rapid Development System easy to use. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 15. Using the Rapid Development System would be pleasant. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 16. I plan to use the Rapid Development System in next six month. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 17. I Learned a great deal of information about control design/insertion from this week's lab 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

18. I learned a great deal of information about control design/insertion from class lectures 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 19. I learned a great deal of information about control design/insertion from the class text 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

Page 15.541.25

20. I found this week's lab on control design/insertion to be very motivational 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 21. I found the class lectures over control design/insertion to be very motivational 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

22. I found the class textbook's coverage of control design/insertion to be very motivational 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

23. This week's lab activity over control design/insertion was applicable to "real world" engineering 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

24. The class lecture over control design/insertion was applicable to "real world' engineering 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 25. The text book coverage of control design/insertion was applicable to "real world" engineering 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

Page 15.541.26

26. Before the lab activity that covered control design/insertion, I knew a great deal about the subject 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 27. After the lab activity that covered control design/insertion, I knew a great deal about the subject 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

28. I would like to belong to a science club.

1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

29. I would dislike being a scientist. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 30. Science lessons are fun. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 31. I get bored watching science programs on TV. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 32. I would like to work with people who make discoveries in science. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

Page 15.541.27

33. I dislike science lessons. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 34. I would like to be given a science book or a piece of scientific equipment as a present. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 35. I would dislike a job in a science laboratory. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 36. School should have more science lessons each week. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 37. I dislike reading books about science in my leisure time.

1 2 3 4 5 6 7

Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 38. Science lessons bore me. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 39. Working in a science laboratory would be an interesting way to earn a living. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree 40. Science is one of the most interesting school subjects. 1 2 3 4 5 6 7 Strongly Disagree Somewhat Neither Agree Somewhat Agree Strongly Disagree Disagree nor Disagree Agree Agree

Page 15.541.28

APPENDIX 3: Eye Tracking Terms16: Gaze: An eye tracking metric, usually the sum of all fixation durations within a prescribed area. This can be represented using a Gaze Plot. Heat Map: Aggregated gaze data presented using heat maps with darker color showing increased fixation and lighter showing decreased fixation. Area of interest: Researchers define areas of interest over certain parts of a display or interface under evaluation, and analyze only the eye movements that fall within the defined area. Saccade: An eye movement occurring between fixations, typically lasting for 20 to 35 milliseconds. The purpose of most saccades is to move the eyes to the next viewing position. Visual processing is automatically suppressed during saccades to avoid blurring of the visual image. Scanpath: An eye-tracking metric, usually a complete sequence of fixations and interconnecting saccades.

Page 15.541.29