Prototyping and Evaluation

8/4/2019 Prototyping and Evaluation

1/23

Prototyping and Evaluation Final

Project Planning

With the given chart below we will have our initial prototype hand drawn by April 13th.

What this will allow us to do is meet with Todd and review the low fidelity screen shots

to draw more feedback from Todd. We believe this is a better approach then having

something sketched up because this will allow Todd to be more critical and influence

the design because he will feel like it will be easier to change.

Following this guide line the initial sketches to be done on the computer will be

completed by Monday April 18th to be presented to the class.

Goals Due

Dates

Lead the

specific partof the

project

Who / When feedback will be given

Review Case

Studies

April

13th,

2011

Albert Case studies will be written by each

team member. After the due date

assigned by our team we will then be

given 3 days to review each case study

reviewed.

BrainstormingMethods and

Materials

April13th,

2011

Mike All members will have been given anequal voice due to the chat log and

associated brainstorming sessions.

Initial Prototype April

13th,

2011

Sarah Feedback will be given by Sarah as

Albert will be drawing the sketches

Usability

Specifications

April

13th,

2011

Matt Mike will review the usability

specifications as he has written them in

the past and is familiar with this.

Designing a

Usability Study

April

15th,

2011

Mike Sarah will review the study by April

18th.

Planning and April Albert Albert will review this on April 20th to


2/23

Communication 18th,

2011

make sure we are well prepared.

Model-based

evaluation

April

24th,

2011

Sarah Sarah will review this by April 26th.

Pilot Testing April

24th,

2011

Matt Matt will head the pilot testing for the

usability study written earlier by April

26th.

Conducting a

Usability Study

April

26th,

2011

Mike Mike will find participants to review our

product by April 28th.

SummarizeUsability Study

Results

April30th,

2011

Albert Albert will take the data as a non-biasmember and review it and analyze

results by April 31st.

Managers -Albert Knowingbeforehand who will schedule the meetings, interview

the clients, take notes, and begin to develop the scenarios and claims will help us

by saving time for when we actually meet. With consistent roles we no longer

have to assign people every time we have something to do. Lastly assigning

these roles will give our team a way to place accountability on each member and

ensure there is no slack in our group. Planning manager(such as schedule meetings) - Albert will be in charge of

planning what it needs done every meeting. Create a detailed plan so everyone

is on the same page about what needs to be done. Spread out all deliverables so

members will do a little every day or so. Make sure we focus on the roles more in

design phase and work on improving weaknesses.

Productivity manager(developing maintain scenarios) - Sarah will be in charge of

productivity. She will be responsible for making sure we are on task, meet

deadlines, and everyone is focused on their roles.

Communication manager(taking notes) - Matt will in charge of communications

and will be in charge of developing equity in verbal participation and building onideas.

Mediation manager(interviewing clients) - Albert will be the primary mediator

between the client and our team. He will be in charge of ensuring the team and

our client are all on the same page

Critical Evaluation (Refactoring Claims) - Mike will be responsible for critically

evaluating the work done by our individuals and making sure that Todd is happy


3/23

with the work produced. He will be responsible for solving conflicts when

necessary. Work on asking people to provide usability rational to defend ideas

and focus on identifying claims and tradeoffs.

Feedback from Mike will be given in a 24 hour window upon any work being completed.

From this Mike will be in charge of critically evaluating and providing feedback to each

individual team member.

Model-based evaluation

Overall Goal: Building a Virtual Kid from start to Finish

1. Sub - task: Adding Personal Information.

a. Will the user try to achieve the right effect?

i. From the home page the user is immediately prompted to begin tobuild the virtual kid. Upon clicking Create Virtual Kid from the left

hand bar they are given the basic attributes to enter the basic

information about the VK.

b. Will the user notice that the correct action is available?

i. No they may be confused upon finishing the creation if done

successfully. This is because there is no success notification on the

add food page telling them they added their information correctly.

c. Will the user associate the correct action with the effect that user is trying

to achieve?

i. No, they may be confused as to what exactly they are adding until

they complete the full process

d. If the correct action is performed, will the user see that progress is being

made toward the solution of the task?

i. No, there needs to be a process bar stating where they are in the

process and giving them perspective on what needs to be done to

finish.

2. Sub - task: Adding Foods to the Virtual Kid

e. Will the user try to achieve the right effect?

i. Not yet. With the low fidelity of the prototype it is unclear for theuser on how he should be dragging and dropping the food onto the

plate. Perhaps in the next version it should give more tip boxes on

the dragging and dropping of the food.

f. Will the user notice that the correct action is available?


4/23

i. No, again because of the low fidelity the user can not complete any

of the tasks at this time. However with the proposed layouts it

should be obvious you are essentially creating a meal at this time.

g. Will the user associate the correct action with the effect that user is trying

to achieve?

i. Yes. We believe that a plate with pictures is a good metaphor for

the creation of a meal plan. Much simpler then typing in food items.

h. If the correct action is performed, will the user see that progress is being


i. No, again there is no progress bar that needs to be added.

3. Sub - task: Adding Work Outs to the Virtual Kid

i. Will the user try to achieve the right effect?

i. Currently if the user can figure out the adding of food, the adding of

exercises is much similar and should be simple and easy to figure

out. However finding the page may be difficult because there is notyet a progress bar to view what they need to do before building

their Virtual Kid.

j. Will the user notice that the correct action is available?

i. Yes, the user should be able to tell from a calendar metaphor with

pictures of specific exercises allowing the individual to not even

have to read to discover what this task is trying to accomplish.

k. Will the user associate the correct action with the effect that user is trying

to achieve?

i. Yes, it should be clear to the user that we are trying to build a

virtual health plan and adding exercises to this plan is directlyrelative.

l. If the correct action is performed, will the user see that progress is being


i. No again there needs to be a progress bar.

Design Changes

Based on this model prototype evaluation a few design changes quickly standout to our

team that need to be incorporated. The most notable ones that should be fixed before

developing any further are as follows:

1. Progress Bar: The user tends to get lost in where they are in the application and

there needs to be a way to tell the user how and where they are to proceed in the

timeline of creating a virtual kid

2. Drag and Drop: The drag and drop features are currently inactive and will not

allow the user to receive an example of how the prototype is to act. This may


5/23

need to be changed to allow for dragging and the creation of food to truly see

how simple this user interface is.

3. Basic information: Upon adding basic information to your virtual kid you are not

prompted at all that this information is correct, and that it has been stored to your

Virtual Kid. This is once again partially due to the low-fidelity of the application

and due to the fact that the kid is not actually stored at this point yet.

Pilot Testing

Beginning Survey

Thank you for agreeing to participate in the study. Before we begin, it will be useful for

us to know a little bit of background information before we observe your experience with

the Virtual Kids application. Throughout this study it will help us gain a better

understanding of our own application through your interactions.

Name: _Chima Nwanna______ Occupation: __Student_____

Age: _20_ Years of Education: ___up to junior year of college___

How often do you use a computer: _40__hours/week

Please describe your typical computer use (e.g. over a period of a week)

Use various emulators, watch movies, facebook, computer programming

Do you have any experience with nutrition or other proper exercise habits?

No

Is there anything else we should know about you before beginning the survey?

n/a

Instructions

In the next 30 minutes or so, you will be carrying out 4 tasks with the Virtual Kids

application. You will primarily work alone and be responsible for completing these tasks

as quickly as possible. As you go through the application talk allowed and what you are

thinking so that we can follow along as you use our product. We will give you a basis

story that you are to achieve and will leave parts out on purpose so that you will be leftwith figuring some key parts of our application once. We will intervene and help you

along the way if needed.

At the start of each task, please say out loud: Beginning Task followed by the name of

the task you are about to complete. It is very important for us to know how long you take


6/23

with each task and what you are thinking as you go through them. Feel free to ask any

questions as you go along and complete the tasks.

To begin, read the scenario and task list below to gain some background information of

the context of the task list. Then, when ready, click the link under Prototype to get to

the prototype page. You can keep referring back to these two while using the system.

Scenarios

You are a 10 year old whose teacher is conducting a class on Virtual Kids to see how

healthy your current diet is, and what path that diet will lead you towards. The teacher

starts off the class by having them create a virtual kid

You are a 12 year old kid who has just come home from school. You had a class today

that taught you about nutrition and learned that there can be a lot of unhealthy things in

your diet. The virtual kid software was recommended in school for you to learn what is

good for you and bad. You also want to check out exercises that will benefit you most.You go on the computer and find the software.

Task List

1: Successfully create an account and log into the virtual kids system

2: create a virtual kid

3: plan meals according to your current food consumption for a typical week

4: learn about benefits of healthier foods and change your meals accordingly to see a

healthier looking virtual kid

Prototypehttp://www.virtualkids.weebly.com

Post Survey

Name: ___Chima_____

What did you like the most? And why?The helper option on the food plate (if it worked)

What did you like the least? And why?The gap in space between the menu and the rest of the screen.


7/23

If Virtual Kids were made available to you would you use it? And why?Probably not, Im not in the target demographic though

1. Using the Virtual Kids software gave you a better understanding of your specifichealth?

Strongly Disagree Disagree Neutral Agree Strongly Agree

2. Would you use the Virtual Kids software on a daily basis to record your nutrition?


3. Learning about the nutritional levels of various foods will help me make mydecision on eating choices from now on


4. After learning about various exercises that may be more beneficial than othersare you more likely to participate in higher intensity activities.


5. Seeing the various aspects of the food pyramid on a plate do you now have abetter understanding of what should be eaten from a typical meal?


Do you have any suggestions for the overall use of the application that you think thedevelopers should be aware of?Nope

Thanks again for your participation!Pilot Testing

Due to the requirements of having at least two team members, we were limited in our

selection. We wanted to use children between 5th and 8th grade as they are the future

users of our system, but there was no time when we could have at least 2 team

members present and someone that young. Instead we used Matts roommate, Chima,who is a computer engineering student. Since his majors building is connected to ours

it was easier to find a time that suited all of us.

Notes

Scenario was good. It seemed that the tester understood exactly what was expected.

Task 1: Successfully create an account and log into VK system

27 seconds


8/23

This task had to be coupled with task 2. Our prototype is only partially

functional, and we didnt realize when we first made the task list that our

create an account and create virtual kid are essentially the same

process. With more functionality this will change.

Task 2: Create a Virtual Kid

See Task 1 description Chima was confused because of lack of

functionality in prototype

Task 3: Plan meals according to your food consumption for a typical

week

43 seconds

Again, Chima was confused because lack of functionality in the system

and the wording of the task. We should change this to reflect the current

level prototype we have

Task 4: Learn about benefits of healthier foods and change your meals

accordingly to see a healthier looking virtual kid73 seconds

The word Learn was too vague for Chima, he did not know when he

should stop reading about new foods

Errors

Chima did not have any errors with our system, as there arent enough wrong choices to

make at this

level of prototype. He did however have confusion as was noted in the task by task

break down.

Procedural Changes

Based on the results of the pilot test, we now recognize the need to accommodateusers due to our

prototype only being partially functional. Because of this, we will break the tasks down

into a

descriptive list for the test subjects. This list will utilize instructions that ask for specific

tasks rather than

broad ideas. Initially, we embraced these ideas because we want kids to explore, thats

the point of our

system. But at this stage it is not feasible. The new task list reads:

1) Successfully create an account and you will automatically log on to the Virtual

Kids system for the first time

2) Press the + button next to steak. This adds a steak to your already partially

completed plate.

3) Go to Exercise menu, and select Add exercise. This brings you to the exercise

page. Click biking on the left hand side of the page to see more about it.


9/23

4) Go back to food by selected Meal Plan, and examine foods. Now click the

chicken to see a simplified version of the chicken nutrition facts.

Conducting a Usability Study

Usability Test:

The bottom 3 pages are what we present to each user. This top page is for our notes.

We compiled it into a single file per person for ease of use.

Intervention Plan

Our path of execution is very descriptive as we have only a partially functional

prototype. If the user strays off this path, we will intervene and explain that they need to

perform the tasks in order. Other than that, we will observe user errors instead of

intervene as they are important for our analysis.

Task Notes



Time: 22 seconds everything went smoothly

Thinking aloud: click create, typing, submitting

6) You are now on the food plate screen. Press the + button next to steak. This

adds a steak to your already partially completed plate.

Time: 8 This user moved through everything very quickly, despite having one

of the lower amounts of computer usage.Thinking aloud: Clicking plus



Time: 18 seconds Remembered the navigation bar at the top so went there

first unlike other users

Thinking aloud: clicking exercise and I see biking so Ill click on it.



Time: 11 seconds

Thinking aloud: Back up to the menu at the top and clicking examine foods,

clicking on chicken for more info


10/23

Usability Test:

The bottom 3 pages are what we present to each user. This top page is for our notes.

We compiled it into a single file per person for ease of use.

Intervention Plan

Our path of execution is very descriptive as we have only a partially functional

prototype. If the user strays off this path, we will intervene and explain that they need to

perform the tasks in order. Other than that, we will observe user errors instead of

intervene as they are important for our analysis.

Task Notes



Time: 32 seconds most of time spent typing in info

Thinking aloud: Create, submitting.10) You are now on the food plate screen. Press the + button next to steak. This


Time: 16 seconds didnt recognize the steak right away.

Thinking aloud: Reading through the page to find the word steak somewhere..

and pressing plus button



Time: 23 seconds Like a few other users he couldnt find the exercise menu

because of the large image between the rest of the site and the drop down

menu for exerciseThinking aloud: Wheres exercise? Wheres exercise Clicking Exercise,

adding Exercise, and examining biking

Error time: took about 15 seconds to find exercising



Time: 10 seconds Moved through very fast at this point

Thinking aloud: I saw meal plan next to exercise, click examine food and now

Im checking the nutrition facts


11/23

Beginning Survey





Name: _Dan Wittmer___ Occupation: __Student - Kinesiology__

Age: _20__ Years of Education: __Junior__

How often do you use a computer: __12__hours/week


watching movies, facebook


I like to lift and run but not much more than that. I have a balanced diet except on

weekends.


n/a

Instructions


application. You will primarily work alone and be responsible for completing these tasksas quickly as possible. As you go through the application talk aloud and what you are


story that you are to achieve and will leave parts out on purpose so that you will be left

with figuring some key parts of our application once. We will intervene and help you






To begin, we will read you the scenario and task list below to gain some background

information of the context of the task list. Then, when ready, click the link under

Prototype to get to the prototype page. You can keep referring back to these two while

using the system.


12/23

Scenario







good for you and bad. You go on the computer and find the software.

Task List





3) Go to Exercise menu, and select Add exercise. This brings you to the exercisepage. Click biking on the left hand side of the page to see more about it.



Prototype

http://www.virtualkids.weebly.com

Post Survey

What did you like the most? And why?

It was easy like applications should be.

What did you like the least? And why?n/a

If Virtual Kids were made available to you would you use it? And why?Probably not, I know this stuff already and I dont like computers

6. Using the Virtual Kids software gave you a better understanding of your specific

health?





13/23







Do you have any suggestions for the overall use of the application that you think the

developers should be aware of?I think there should be a way to remove food from plates rather than just add.Thanks again for your participation!

Beginning Survey Tester 2





Name: _Chris Payne___ Occupation: __Electrical Engineering__

Age: _20__ Years of Education: __Junior__



twitter, YouTube


No

Instructions



as quickly as possible. As you go through the application talk aloud and what you are


14/23












using the system.

ScenarioYou are a 10 year old whose teacher is conducting a class on Virtual Kids to see how







Task List

5) Successfully create an account and you will automatically log on to the VirtualKids system for the first time







Prototype


Post Survey

What did you like the most? And why?It overall just seemed well thought out. It was also colorful so I think it will be good forkids


15/23

What did you like the least? And why?You have to go to a separate page to examine foods than the one that puts it on yourplate.

If Virtual Kids were made available to you would you use it? And why?Probably not, I couldnt motivate myself to add food to a list everyday online.











Do you have any suggestions for the overall use of the application that you think thedevelopers should be aware of?Just the thing about the separate pages I mentioned earlier.Thanks again for your participation!

Beginning Survey Tester3





Name: _Dan Gonzalez___ Occupation: __Student Food science__


16/23

Age: _21__ Years of Education: __Junior in college__



facebook, aim, music


Not really, I would like to though


no

Instructions

In the next 30 minutes or so, you will be carrying out 4 tasks with the Virtual Kidsapplication. You will primarily work alone and be responsible for completing these tasks








with each task and what you are thinking as you go through them. Feel free to ask anyquestions as you go along and complete the tasks.




using the system.

Scenario









17/23

Task List









Prototype


Post Survey

What did you like the most? And why?I liked how it was very picture oriented because it made it easy to understand

What did you like the least? And why?I cant click the back button in the browser because its inconvenient

If Virtual Kids were made available to you would you use it? And why?Possibly, but only if I can use the back button





18. Learning about the nutritional levels of various foods will help me make my

decision on eating choices from now on





18/23

20. Seeing the various aspects of the food pyramid on a plate do you now have a

better understanding of what should be eaten from a typical meal?


Do you have any suggestions for the overall use of the application that you think thedevelopers should be aware of?You cant use the back button

Thanks again for your participation!

Beginning Survey Tester4





Name: _Derrick Armold___ Occupation: __Engineering Intern__

Age: _21__ Years of Education: __Junior in college__



email, facebook, matlab


Yes, Ive been a wrestler up through high school and Im not on the penn state boxing

team. So I know a lot about exercise and I also learned to eat a balanced diet while

cutting weight to get into a lower weight class.


I dont normally use computers unless I need to.

InstructionsIn the next 30 minutes or so, you will be carrying out 4 tasks with the Virtual Kids







19/23









using the system.

Scenario








Task List




adds a steak to your already partially completed plate.15) Go to Exercise menu, and select Add exercise. This brings you to the exercise




Prototype


Post Survey

What did you like the most? And why?I liked the simple layout. And also how I could get to everything I needed from the dropdown menu at the top.

What did you like the least? And why?I couldnt always see the drop down menu from the rest of the site.


20/23

If Virtual Kids were made available to you would you use it? And why?Probably not, I know this stuff already and I dont like computers





23. Learning about the nutritional levels of various foods will help me make my

decision on eating choices from now on






Do you have any suggestions for the overall use of the application that you think thedevelopers should be aware of?NoThanks again for your participation!

Summarize Usability Study Results

Intro

Due to the necessary requirement of having at least two team members present during

all usability testing, we were unable to get younger kids to test our prototype. The only

place any of us knew anyone who fitted the target demographic (5 th-8th graders) was at

home where at most, only one group member could be available. Instead, we had

Matts roommates be out testing subjects. We first had one roommate perform the pilot


21/23

test of our system, then made changes accordingly, and then gave the new and

improved usability tests to the remaining four roommates.

Background

Usage in hours/week

Name Age ComputerUsage

Derrick 21 15

DanW 20 12

DanG 21 20

Chris 20 14

Summary of Performance- Time/Error Data

This is the time for each task in seconds, and includes error times. The users go in

order of experience level, based on their computer usage per week (table above)

Name ExperienceLevel Task1(time) T1errortime Task2(time) T2errortime Task3(time) T3errortime Task4(time) T4erDanW Low 32s 0 16s 0 23s 15s 10s

Derrick Medium 28s 0 12s 0 36s 20s 13s

Chris Medium 22s 0 8s 0 18s 0 11s

DanG High 24s 0 10s 0 21s 0 n/a(seetasknotes) n/a

The N/A recorded for task 4 times and error time was due to a necessary intervention.

Dan G clicked the back button of the browser when told to go back to a page he had

been on before. Unfortunately in our prototype the only way to properly navigate from

page to page is through the menu bar.

Name ExperienceLevel AvgTaskTime AvgErrorCount

DanW Low 20.25s .25/task

Derrick Medium 22.25s .25/task

Chris Medium 14.75s 0/task

DanG High 18.33s .25/task

Summary of Reaction/Satisfaction Ratings (tabular)

On our report we had a scale of strongly disagree strongly agree. We converted it to

a scale of 1 to 5 for quantitative purposes:

Name ExperienceLevel Q1 Q2 Q3 Q4 Q5 Average

DanW Low 2 3 4 2 2 2.6Derrick Medium 3 2 4 3 3 3

Chris Medium 4 2 3 2 3 2.8

DanG High 3 4 4 3 4 3.6

Average 3 2.75 3.75 2.5 3 3 These satisfaction ratings show a slight correlation with the amount of time a person

spends on a computer a day with their overall ratings of our system. This makes sense


22/23

but we think we should make it as close to even as possible, because we dont want to

exclude any users based on lack of knowledge.

Descriptive Summary of Qualitative Data

Here is a summary of the think aloud comments. When relevant, the level of user is

noted.

Task NoteworthyComments

1 nothinginteresting,everyonesaidsomethingaboutclickingcreate,enteringinfo,and/orpressingsubmit

2 I'mlookingforsteakbutIdon'tseeit-Unrelatedtolevelofcomputerusage

3 AllIseeisstuffaboutfood,notaboutexercise-Alldifficultylevels

Irememberthemenubaratthetopinthebeginning-Mediumuser

4 Irememberfromaprevioustaskthateverythingisatthemenubaratthetop-MediumandAdvancedus

We get a lot of the same comments out of most people, and there is usually one

comment at most that strayed from the other 3 testers.

Our surveys had numerous questions that gave us insight to data that couldnt berepresented with numbers. They included questions about the testers favorite and

least favorite aspects of the system, as well as asking if there is anything else they

would like to mention to us that wasnt included in the other questions.

Least Favorite: Nearly everyone had the reaction comment that the menu at the top of

the screen was too far from the rest of the prototype. They couldnt see it in the same

browser window! We are in the process of attempting to change this, unfortunately

since our prototype was built using a template with a static header image size, it is

difficult.

Favorite: Everyones favorite part of it was along the same lines (colorful, picture

oriented, simple). This is what we were striving for and we are glad that it had the

intended effect on users.

Do you have any other comments?: Most users said no, but one user (the advanced

one) did not like how the back button in the browser was unusable. Unfortunately this is

another limitation of our prototype. The only other user response is actually a good one

we didnt think of. The least experienced user suggested we create a way to remove

food from a plate rather than just add to it. This was an oversight of ours and we are

glad he spotted it!

Interpretation of Usability Study Results

For the most part, our team expected the tasks given to be completed in a manner ofease. New concerns that were brought to our attention were the menu being too farfrom the rest of the prototype and task four being too broad of an explanation. This hadto be changed to include more information of what users were to do in this task.


23/23

Interesting results found was task three was most difficult for all users to complete. Onaverage, all users took the longest amount of time to complete this task. The task wasto add the biking exercise to an exercise journal. There were some interestingdifferences between specific users. Dan G, who was of a high level of computer use,completed his tasks on an average of 18.33 seconds. Chris P, of medium level

computer usage, completed his tasks on an average of 14.75 seconds. This is aninteresting difference because based on the amount of hours (20) that Dan works on acomputer weekly compared to Chris P who spends 14 hours weekly on a computer.Chris, surprisingly, on average, completed tasks at a faster pace than Dan who hasmore computer use weekly. Another interesting difference between users is the widerange of average completion of tasks between low and medium experienced users.Derrick, who uses a computer on average of 15 hours per week, completed tasks onaverage of 22.25 seconds. Dan W who uses the computer on average of 12 hours perweek (low) has an average of 20.25 seconds. The numbers are so close even thoughthe users differ in amount of time spent using a computer. One change made, beingvery necessary, was the wording of task four. Originally, task four was worded "Learn

about benefits of healthier foods and change meals according to a healthier lookingvirtual kid." The word "learn" is too vague. We then changed the task to be worded as"Go back to food by selecting 'Meal Plan' and 'Examine Food'. Now click the chicken tosee simplified version of chicken nutrition facts." After changing the wording of this task,it took Chima 73 seconds to complete the task, after being changed it took users anaverage of 11 seconds to complete the task. Some changes we can make after theusability testing is the spacing between the top menu bar and the rest of the prototype.Since this is not viewable in the same browser we would change this. Overall, usersenjoyed the picture orientation and how it helped them to understand areas of theprototype. The simplicity of the prototype helped tested users to navigate throughout.

Prototype

http://virtualkids.weebly.com/index.html

Documents

Prototyping and Evaluation