Pass 2 By pWave

Catherine Lee (32556136)

Robyn Castro (37283141)

Nicholas Leung (30252143)

David Wong (15314123)

Howard Zhou (48986137)

Table of Contents

A1. Redesign Rationale 3

A2. Pass 2 Plan and Evaluation 4

C1. Subjects 9

C2. Evaluation Results 10

C3. Final Design Rationale 12

C4. Design Process Reflection 13

D. Resource Management 14

Appendix A: Study Instruments 16

Appendix B: Raw Data 19

Appendix C: Sample Information and Consent Forms 19

Appendix D: List of References 33

A1. Redesign Rationale

A clear, defined HCI problem must be identified to create a successful project. When creating our previous project, we did not have a clear idea of who our users were. We had an age range of users, but we were not aware of their limitations and needs, and therefore could not identify what tool could be designed to support an unclear group of users. While we could have possibly kept the concept from our previous project, memories, there was very little of our project that could be kept. Our user group was unclear. We did not have a defined need or problem from the vague user group, and the solution proposed was generic. Without talking to real users who live with these conditions, it would be difficult to identify problems they would like a solution to, and these users may be difficult to find to interview. Because of this, we decided to instead start by identifying a problem. Speaking to students in the University of British Columbia, we discovered some clear issues that UBC’s Degree Navigator tool suffers from, and decided to pursue a redesign of the tool instead. We decided to talk to users first to understand problems before designing a solution. We found that when users use Degree Navigator, they are provided with uninformative screens and many options that are not relevant to their needs. For example, a student who wants to view their progress in their degree does not understand what an “Arts Minor Report” is for.

→

Figure 1. A depiction of screens a user sees when selecting a report in Degree Navigator.

Also when students wanted to determine what courses they can take to complete unfulfilled requirements in their degree, information provided upon clicking on a requirement were generic (figure 2) or did not clearly indicate which courses the student has not yet taken (figure 3).

Figure 2. Generic course requirements Figure 3. Detailed course requirements

Students who were interviewed expressed their dissatisfaction with the irrelevant options presented to them and lack of information in planning future courses to take. In conclusion, many students struggled to comprehend how the information from Degree Navigator fulfilled their needs, and may give up and use alternative resources such as a faculty website or consult a faculty advisor. A2. Pass 2 Plan and Evaluation

Because students struggled to make sense of the information in Degree Navigator, we will focus on improving understanding of the relevance of the information on the tool to their degree: Goal: Improving the Degree Navigator tool so that users will be able to understand how the information provided is relevant to their degree.

Evaluation: We will conduct a controlled study to measure how well users understood how they could information relevant to their degree. This study will include tasks with quantitative measures to evaluate their understanding, and qualitative interview questions to better understand their actions and thoughts. More detail will be provided in section B1 and Appendix A.

To perform these tasks, we will need a mid-fi horizontal prototype so that users can perform

simple and advanced tasks on both our tool and the original Degree Navigator. This prototype

will need to provide users with enough information to create a plan to complete a hypothetical

degree. A reasonable prototype fulfilling these requirements can be created using Axure.

Additional Analysis and Evaluation We performed a cognitive walkthrough to analyze our low-fidelity prototype before proceeding

to a higher fidelity. Issues identified may be addressed in the mid-fi. We asked ourselves the

following four questions while attempting each task supported in the the low fidelity prototype:

a) Will the correct action be evident to the users?

b) Will users connect the correct action with their goal?

c) Will users interpret the system’s response to the chosen action correctly, i.e., will users

know from this response, whether they made the right or wrong choice?

d) Will users’ mental models be affected? Will new concepts be added, or existing

concepts lost?

1) View Program Requirements

a) Yes, upon entering the application the program requirements can be seen

b) Yes

c) Yes, they will be able to see the requirements

d) No

2) See Progress Towards Degree Completion

a) Yes, a progress bar with the number of credits completed is shown

b) Yes

c) Yes

d) No

3) See which courses can fulfill a specific requirement

a) Yes, a downwards arrow is a common indicator of a drop-down menu, although

the style of the arrow should be changed to a more common one

b) Yes

c) Yes, the appropriate courses and/or sub-requirements will show up

d) No

4) See course prerequisites / corequisites

a) Sometimes. For specific courses, the text is highlighted, making hovering an

obvious action. However, not all courses can be hovered over.

b) Likely, hovering over something is usually associated with more information

c) Yes, the prerequisites and corequisites will show

d) No

5) See course descriptions

a) Same as 4 a).

b) Same as 4 b).

c) Not necessarily, as the descriptions are on the SSC, and only a link is provided.

d) Yes, traveling to another site is required to view course descriptions.

Annotated Designs - Low Fidelity Prototype

We created a simple low fidelity prototype to resolve some of the issues we found. Users are

shown figure 4 upon logging into the new Degree Navigator. A few more details are written in

the prototype itself. Users can press on the down arrow to expand each progress bar for a full

list of courses, like the third year courses shown. Users are shown figure 5 when they hover

over a course they can choose.

Figure 4. Low Fidelity Prototype Main Screen

Figure 5. Hovering over a course in the Low Fidelity Prototype

B1. Evaluation Protocol

Background & User Requirements

Student wants to know the progress of his/her academic program: ● the credits (acquired, total) ● the requirements (mandatory courses, electives) ● what courses fit into what requirements

A tracking tool should be designed for students to:

● understand or have general ideas about their program progresses ● have a plan of what courses need to take to fulfill the unsatisfied requirements

Problem Statement: There is no online tool that allows all UBC undergraduate students to

reliably track their progress in completing their degree.

Definition of Reliability:

● The users should get the correct information regarding their program statuses and requirements.

● The users should be capable of understanding this information. ● The users should be confident in using this information to choose courses correctly and

monitor their progresses. The tools available in UBC are:

● Degree navigator - is not reliable because information isn’t presented in an obvious manner students can identify with.

● Faculty Advisors - need to schedule meeting times at UBC ● UBC SSC - insufficient information; students use it mainly for checking course grades

Due to Degree Navigator’s unreliability, many students do not trust Degree Navigator and cannot locate specific information they want to know, and believe that Degree Navigator contains insufficient information. Students who feel that the tool is unreliable and difficult to understand will go take extra time to schedule an appointment with their faculty advisor to determine their progress in completing their degree. Type of Study: Controlled Experiment

Purpose: Compare the Degree Navigator with the Degree Tracker to show, with statistical significance, remarkable reductions of errors and time made in tracking the program progress using the tool, and a higher user satisfaction. Independent Variables:

● Degree Navigator version (Discrete)

○ Degree Navigator

○ Program Tracker

● Tasks To Complete (Discrete)

Dependent Variables

● Completed tasks

● Task completion time

● User Satisfaction (Likert Scale)

Nuisance Variables (Mitigation in Brackets)

● Familiarity with the original degree navigator

● Familiarity with degree requirements

● Student faculty/major

● Technical background in using online tools

● Order Effect (change order of independent variable testing)

Subject Pool

Our subject pool will consist of people that represents each student cohort. We will have 7

different subjects participating in this study. These include current and former students of UBC

with an interest in pursuing a future degree at UBC.

Hypothesis Testing

● Completed tasks

○ Null Hypothesis - There is no difference in completed tasks made in Degree

Navigator and completed tasks made in Degree Tracker.

○ Alternate Hypothesis - There are more completed tasks made in Degree Tracker

than user errors made in Degree Navigator.

● Task Completion Time

○ Null Hypothesis - There is no difference in task completion time in Degree

Navigator and task completion time in new Degree Tracker.

○ Alternate Hypothesis - It takes less time to complete tasks in Degree Tracker

than it takes to complete tasks in Degree Navigator.

● User Satisfaction

○ Null Hypothesis - There is no difference in user satisfaction in using Degree

Tracker compare to using Degree Navigator.

○ Alternate Hypothesis - There is a higher user satisfaction in using Degree Tracker

than using in Degree Navigator.

C1. Subjects

Our subjects are students of UBC who would want to use Degree Navigator or a similar tool to

plan for the completion of their degree. Ideally, our subjects have some familiarity with Degree

Navigator, and use the tool only a few times a year around course planning and dropping times.

However, this is not required, and there are students who are the target audience of this tool

without these characteristics. All UBC and former UBC students who may have an interest in

obtaining another degree from the university therefore qualify for this study.

The users tested include two former UBC students who expressed interest in possibly obtaining

a degree from UBC in the future. One was a former Computer Science major and another a

Biochemistry major. The Biochemistry major would use Degree Navigator a few times a year,

but often gave up and turned to the advisor, so an improved system that he can better

understand will be beneficial to him. The Computer Science major did not use the original

Degree Navigator during his degree, and instead referred to his faculty’s website for course

information. He expressed that such a system would have been useful if he was able to locate it

in among the tools available to him as a student.

The users tested also include a recently-graduated Electrical Engineering student and an Art

student with a double major of Economics and Statistics. The graduated student used the

degree navigator 1-2 times per academic term mainly for confirming the match between a

course and a requirement the course can fit, and he complained that the degree navigator was

not very helpful in solving this problem . The Art student uses degree navigator once per

academic term and he complains that the credit assignment done by the degree navigator does

not follow his expectations.

The rest of the experiment subjects are Computer Engineering students who uses degree

navigator 1-2 timers per academic term, but they also develop their own methods to track their

program’s progress and use degree navigation for course confirmation purposes.

The subjects we interviewed in our resubmitted Pass 1 fit the profile of users in our controlled

study. However, the same subjects were not used, because they were asked to perform a few

tasks similar to the controlled study to help us determine the issues with Degree Navigator.

Consequently, the individuals interviewed for Pass 1 would be too familiar with the tasks in the

interview, providing a biased measure of their understanding original system. We selected new

subjects to ensure that their knowledge would not affect the study.

C2. Evaluation Results

In our “controlled studies”, we evaluated a user’s understanding of how the information helps

them determine their progress in completing their degree. Since “understanding” is a rather

broad concept, we had our subjects complete tasks in both Degree Navigator and Degree

Tracker.

For each, we noted whether or not they were able to correctly complete the tasks (ie. identify

all unsatisfied requirements successfully). We also timed the users. While time is not directly

linked to understanding, completing a comprehension task in a shorter period of time is

another indicator of understanding. Each subject was also asked to rate their experience on a

likert scale from 0 to 5. 0 meant that they found the task challenging, difficult to comprehend,

and overall unpleasant to perform on the tool. 5 meant that the tool was easy to understand

and a pleasure to perform.

The rate of success was one such factor that users showed improvement in. In terms of success,

several participants had a few tasks they were unable to complete on Degree Navigator.

However, on Degree Tracker, all participants successfully completed most, if not all tasks. This

indicated that participants were able to more consistently understand information presented

on Degree Tracker than Degree Navigator.

After conducting a t-test, on our gathered data we were able to reject the null hypothesis that

there is no difference in success rate between degree tracker and degree navigator, so we

could say with 95% confidence that there actually is a difference. That difference was an

improvement of 2 to 3 more successes when using degree tracker instead of degree navigator

using a confidence interval of 95% confidence.

Unfortunately for time, we had even more trouble reducing nuisance variables, so we used an

80% confidence interval. We felt then that time is not necessarily the best indicator of

understanding, an improvement in time for tasks that require some level of comprehension can

be an indication of a user’s ability to determine relevant information in the tool. Although there

was high variability in the amount of time participants took to successfully identify and

understand information, all users took less time when using Degree Tracker compared to

Degree Navigator. What this really indicates though, is that we did not really perform an

effective controlled study.

After conducting a t-test, on our gathered data we were able to reject the null hypothesis that

there is no difference in task completion time between degree tracker and degree navigator, so

we could say with 80% confidence that there actually is a difference. That difference was an

improvement of 35 to 156 second reduction in completion time when using degree tracker

instead of degree navigator using a confidence interval test of 95% confidence.

Finally, we used satisfaction as a metric to determine how well the users felt they understood

the information. Nearly all participants gave Degree Tracker high ratings, while

satisfaction/comprehension ratings varied from low scores (0-1) to average scores (2-3) with

occasional high scores (4-5).

After conducting a t-test, on our gathered data we were able to reject the null hypothesis that

there is no difference in user satisfaction between degree tracker and degree navigator, so we

could say with 95% confidence that there actually is a difference. That difference was an

improvement of 7.29 to 17.29 s when using degree tracker instead of degree navigator using a

confidence interval test of 95% confidence. In fact we could say that with 95% confidence, a

person would give 7.3 to 18.7 more score to degree tracker compared to degree navigator in

our likert scale survey described above.

To test for variable independence between tool used and their average satisfaction rating we

also used the Chi-Squared test for independence. Which tool they used had two levels: Degree

Navigator and Degree Tracker. Satisfaction rating had three levels: Unsatisfied (0-2), Neutral

(3), Satisfied (4-5). It did not pass the Chi-squared test, so the null hypothesis remains true.

However, we also realised that our sample size was too small for Chi-squared to be statistically

significant. Therefore, no information can be gathered from this.

On top of simple calculations, we interviewed each subject. Raw data and summaries of

responses from each subject is provided as an appendix to this document. Many subjects

verbalized discontent and confusion because of many options available to them in Degree

Navigator. In general, they were pleased to see that Degree Tracker only has a single page, and

indicated that Degree Tracker was significantly easier to understand. This information appears

to match the data we determined in our controlled study.

There are, however, some problems with our study. Due to time constraints, we were only able

to perform our controlled study with 7 participants. We were not able to mitigate all our

nuisance variables either. Time had high variability, since individual abilities affected the time

each person took. Since there were only 7 participants, it was not a representative population,

especially as they were not randomly chosen.

Due to a miscommunication in our team, while one interviewer conducted one interview by

presenting the user with Degree Tracker first. This meant that we did not properly mitigate the

ordering bias.

Furthermore, it is difficult to determine if our study was truly controlled and actually tested.

The measure we took: success/failure/errors, time, and satisfaction/confidence were not well

controlled since the two designs varied vastly and required different pages. Also, participants

were shown the questions prior to conducting a group of tasks, and provided the questions as a

reference. This meant that participants may forget their goal at times and spend some time

re-reading this information.

Our Degree Tracker tool has been found to be appropriate by Science and Engineering students,

but because interviewers used their own Degree Navigator as the “control”. But this may not be

the same for all students, and some errors that may occur as a result of students using their

own degree navigator may not occur. Given this, it is rather hard to say that our data can help

us make a final decision.

C3. Final Design Rationale

After considering our controlled studies results, we do believe that there is a difference

between a students’ understanding of Degree Tracker and Degree Navigator. While the

statistics cannot prove that this is necessarily true, with our findings alone, we can determine

that building an alternative Degree Navigator will likely be a worthwhile endeavor. We are

confident that there is a problem with the existing tool.

Degree Navigator provides information that allows students to determine how close they are to

earning degree. It informs students of what requirements they still need to complete to

graduate. However, because many students struggle to make sense of this information,

especially for more complex tasks like understanding what requirements are unsatisfied and

how they can plan to satisfy them, students will refer to an student advisor’s expertise.

However, because of the weaknesses in our “controlled studies”, while we were able to identify

a difference, we lack a good indicator of whether or not Degree Tracker will significantly help

students. We also don’t know if Degree Tracker is appropriate for all students, given that we

had a small sample size. Given such a low sample size, an 80% confidence interval is low.

Since comprehension is the main aspect we want to improve in creating Degree Tracker, we

also looked at students’ success rate and satisfaction with the tool. In that respect, we are 95%

confident that students are more capable of identifying and comprehending information, and

also feel more satisfied with the information they were given. This gave us some confidence

that our tool does increase some aspects, but due to shortcomings in the controlled studies

again we cannot use this data to make a decision.

If we step back and consider this whole endeavor, if we wanted to fund this project and hire

developers, we cannot say that we can make a final decision. What we did find is that we

learned from this process. We successfully identified a prominent issue that prevents students

from properly understanding their progress towards graduation through UBC’s existing tool.

From here on, the best decision to make would be to refine our study, perhaps with far more

control. The ordering bias should be addressed, but we should also consider what other metrics

we can use to better give us an indication of understanding. Furthermore, the experiment

should be much more strictly controlled to prevent users from re-reading questions if we

intend to use time as a measure. Then, we should pursue a much larger student distribution.

C4. Design Process Reflection

Twelve weeks of CPEN 441 has taught us a lot. Twelve weeks ago, we thought user-centered

design is only about designing an aesthetic user interface while keeping it somewhat functional.

In the span of twelve weeks, we realized that user-centered design is so much more than simply

design an user interface; there are many more aspects one has to take into consideration.

For our team specifically, we learned that identifying a target audience and problems

associated with existing interfaces is particularly important when it comes to user-centred

design. The mistakes we made during the two passes will be a valuable lesson for future

interface design project.

During Pass 1, we designed an photo sharing app targeted towards senior users (around 70

years). However, the unclear target audience made it difficult for us to determine exact needs.

We thought that in order for senior users to navigate the app with as few button clicks as

possible with fewer functionalities. This led to an app looked almost bare-bone, without any

notable usefulness to users. We also picked a difficult group to target; seniors may have certain

diseases limited their abilities, so a design should be catered to those traits. However, It is hard

to involve users when testing an app designed for senior users since many of them have trouble

operating mobile devices and wide variability. A few interviews were conducted and the

anecdotal data collected was not a large enough sample size to draw any meaningful

conclusion.

Learning from our past mistakes, and started with identifying a problem and understanding our

population around it. For pass 2, we redesigned UBC’s degree navigator as the current user

experience of degree navigator is often unsatisfactory. We recognized many flaws in existing

design, but flaws don’t necessarily equate to problems. Not having a clear problem definition

would render new interface design purposeless. The level of user involvement for pass 2 has

increased significantly since a interactive medium fidelity prototype was made before user

interviews were conducted. The design changed slightly after some of the interviews and by

having a prototype, the data collected is less biased and more useful when it comes to deliver

the final design.

Other activities we have to try out during the course includes working with medium fidelity

prototyping tool. It is a valuable experience because in order to make an effective mid-fi

prototype, a designer must stand on the user’s point of view to design an interface while

keeping the problem statement in mind.

However, our mistakes did not end there. When we tried to conduct a controlled study, we

found struggled to find measure that defined understanding and reliability well. Consequently,

our measures (success/failure/errors), a likert scale indicating their level of understanding and

satisfaction, and time were not overly suitable for judging if our tool is a good substitute for the

current Degree Navigator.

We did learn that could determine an issue in an existing HCI interface. What we did not

successfully to was accomplish an experiment that would allow us to draw a conclusion.

D. Resource Management

This section contains the contributions of our team member and the estimated time spent. A

median salary of $26.09/hr [1] is used, as all members are considered design engineers.

Name: Catherine Lee

Hours: 17 hours

Cost: $443.53

Tasks: Preliminary interviews, conducted controlled study on two subjects, rewrote Pass 1,

wrote Pass 2 documentation, participated in team discussions, assisted with mid-fi

development.

Name: Nicholas Leung

Hours: 14 hours

Cost: $365.26

Tasks: New Proposal writing + list of requirements, cognitive walkthrough, pass 2 editing, team

discussions, demo powerpoint, final powerpoint, mid-fi video

Name: Howard Zhou

Hours: 7 hours

Cost: $182.63

Tasks: Wrote Design Process Reflection, created mid-fi

Name: Robyn Castro

Hours: 16 hours

Cost: $417.44

Tasks: Wrote parts of pass 2 report, assisted with redesign, team discussions, study design

assistance, completed calculations

Name: David Wong

Hours: 12.5 hours

Cost: $326.13

Tasks: Designed controlled study plan, interviewed 5 students, wrote follow up information

about interviews in pass 2 report

Appendix A: Study Instruments

Interview Instructions

● During the interview process, the observer need to record the result of the tasks that

are assigned to the participant.

● There are two types of task that the participant need to complete :

○ Type 1 ( T1): A task that involves using a software tool to achieve a certain goal or

to solve an academic problem.

○ Type 2 ( T2): A question seeking an opinion related to a software tool’s

performance or presentation.

● Type 1 task can be classified as Simple (S), M (Medium), H (Hard)

● The observer need to time the participant when doing a type 1 task, and write down the

time that the completion time.

● If the participant can’t complete a type 1 task or gets it incorrect, the observer should

write “0” to indicate failure.

● After you stop timing, ask the participant to rate how easy/difficult it was for them to

complete the task and how confident they feel in their answer. 0 = This was near

impossible to do and I have no confidence in the information I found/couldn’t find the

information. 5 = This was very easy to do.

Pre-Screening Questions

1. How often do you use degree navigator?

2. What’s your program?

Interview Tasks - Degree Navigator

Start Timing

1. Can you identify your program name? T1 (S)

2. Can you identify your total credit required for your program? T1 (S)

3. Can you identify your current credit amount? T1 (S)

Stop timing

4. What do you expect to see once you login into your account? (In other words, what’s

the important information you want to know immediately after login.) Does the tool

fulfill that need? T2

5. Which Degree Description format do you use? Do you use other formats occasionally?

T2

6. Do you like the tool’s presentation format? T2

Start Timing

7. How many unsatisfied requirements do you have right now? T1 (S)

8. What are the unsatisfied requirements? T1 (S-M)

9. For each unsatisfied requirement, how many credits are missing for this requirement?

T1 (M)

Stop timing

Start Timing

10. For each unsatisfied requirement, can you give me some meaningful information based

its name? T1 (S-M)

11. For each unsatisfied requirement, can you tell me what courses (at least 1) will satisfy

the requirement? T1 (H)

End Timing

12. (If the student failed to answer question 11) How do you solve this problem and where

do you look for more information to help you? T2

13. After using the tool, do you have a plan or an idea for your course scheduling in the

upcoming academic year? T2

14. If the student have a plan or an idea, ask the student to briefly explain it. T2

15. (If the student don’t have a plan or idea) What’s your next step? T2

16. Is there any important information that the tool did not tell you? Where will you find

the missing information? T2

Interview Tasks - Degree Tracker

Start Timing

1. Can you identify your program name? T1 (S)

2. Can you identify your total credit required for your program? T1 (S)

3. Can you identify your current credit amount? T1 (S)

End Timing

6. Do you like the tool’s presentation format? T2

Start Timing

7. How many unsatisfied (incomplete) requirements do you have right now? T1 (S)

8. What are the unsatisfied (incomplete) requirements? T1 (S-M)

9. For each unsatisfied (incomplete) requirement, how many credits are missing for this

requirement? T1 (M)

End Timing

Start Timing

10. For each unsatisfied requirement, can you give me some meaningful information based

on its name? T1 (S-M)

11. For each unsatisfied (incomplete) requirement, can you tell me what courses (at least 1)

satisfy the requirement? T1 (H)

End Timing

13. After using the tool, do you have a plan or an idea for your course scheduling in the

upcoming academic year? T2

14. If the student has a plan or an idea, ask the student to briefly explain it. T2

16. Is there any important information that the tool did not tell you? Where will you find

the missing information? T2

Post Experiment Questionnaire

1. How often do you see your academic adviser each year for issues relating to program

progress? Relating to requirement clarification? Relating to course

clarification/satisfaction?

2. For all the issues mentioned above, how many of them should be resolved by the tool?

Appendix B: Raw Data

Our data is provided in Sample-Degree Navigator Analytics.xlsx. Please refer to the spreadsheet

for the raw data and calculations.

Appendix C: Sample Information and Consent Forms

Each subject is briefly described along with a summary of our observations and conversations

with them. This section includes all interview questions conducted. Consent forms will be

provided in addition to this description, but not in any particular order to avoid violating the

privacy of each user.

Subject 1

Subject profile: A former second year UBC student who studied in the faculty of Computer

Science. This student has currently transferred to another post-secondary but intends to return

to UBC for other degrees in the future.

Summary of Observations and Interview Feedback: This student was not particularly aware of

Degree Navigator during his time at UBC. He referred to his faculty website to determine what

courses he should register in during the term, but expressed interest in using the tool if it

provided him with sufficient information for course selection.

When using the original Degree Navigator, he noted that he was not expecting to have been

shown an empty page. He would have expected a report providing information on his

credentials would be shown immediately. The navigation cues of the tool seemed reasonable

for him, and found that in general, the important information like his faculty and the credits he

needs to graduate are not too hard to find. He thinks that the original Degree Navigator tool

would be able to help him plan his courses if he had known about it, but felt that some

descriptions and choices for courses, like Advanced Electives were too vague and difficult to

make sense of.

On the other hand, Degree Tracker shows all the information upfront with a single page. It also

provides specific guides instead of vague descriptions like “Advanced Electives”. He felt that

Degree Tracker did not appear to have the same amount of detail as Degree Navigator, but felt

the additional information on Degree Navigator was not relevant to him.

Subject 2

Subject profile: A UBC alumni who majored in Biochemistry. This former student graduated in

May 2017 and is interested in returning to UBC for a Master degree in the future.

Summary of Observations and Interview Feedback: During his time in UBC, he used the Degree

Navigator tool around 5 times a year, but consulted advisors from the Faculty of Science once

during the year when he could not make sense of Degree Navigator.

This former student felt that Degree Navigator provided too many reports and options. It was

difficult to select the correct report, and if he changed the report version, the number of credits

he successfully completed would change radically. This does not give him much confidence

when completing even basic tasks like identifying how many credits he has completed towards

his graduation, and instead encourage him to seek out his academic advisor. He especially had

trouble differentiating between the UBC Report and Full Progress Reports since they do not

necessary show all unsatisfied requirements. He stated that the inconsistent information on the

tool could lead him to mistakenly think that he fulfilled more requirements than he actually

had, or panic, leading him to consult an academic advisor once again. He also noted that the

tool provides vague descriptions that are too general, like “Breadth Electives”, making it

difficult to determine what courses he can take to complete his degree.

He found that Degree Tracker looked nice and clean. He mentioned that even without a legend,

the colors were nice and easy to differentiate, although perhaps a red color instead of white

would have been more intuitive for incomplete requirements. He felt that the tool was easy to

understand. When asked to determine what courses he could take to fulfill unsatisfied

requirements, he stated that it was quite simple to pick a guide and choose a course.

Subject 3

Subject Profile: The student is a fourth-year Arts student who is completing a Bachelor Degree

in Statistic and Economics in May.

Summary of Observations and Interview Feedback: The student uses degree navigator quite

often in tracking his program status, and he felt that he’s confident in completing tasks during a

short conversation in the pre-screening section. During the experiment, he tended to rely on his

memory in using Degree Navigator to complete the experiment’s tasks, though I already had

assured him that all tasks should be completed base on recognition and observation. It is

possible that the tendency of acting professionally in using tool played a psychological factor in

this student’s mind. When the experiment moved onto the Degree Tracker section, the student

tried to apply the mental model developed from using Degree Navigator to complete the tasks.

Overall, the student completed all the tasks (successfully or not) in a reasonable time.

Since the student only uses Degree Navigator to track his academic progress, he is quite familiar

with the tool in finding information relating to course requirements and gets used to Degree

Navigator’s presentation format. The problem that concerns him the most about Degree

Navigator is the credit assignments that did not work the way he’s expecting. As a result, he is

frustrated by some unused credits that are valid to satisfy some requirements.

The student likes the format of Degree Tracker and believes the tool providing just enough

information to form a simple plan to manage his academic progress. The student made a

recommendation of making some titles more obvious and clarify/define some requirements’

name. In the post experiment questionnaire, the student felt equally satisfied by the tools in

completing the tasks.

Subject 4

Subject Profile: The student is a fourth-year Computer Engineering student who just completed

a year of COOP, and he is taking another year of study before his graduation.

Summary of Observations and Interview Feedback: Surprisingly, this student had never used

Degree Navigator until this experiment. Thus, he completed all the tasks based on just

observations and recognitions on the tool’s presentation format (UBC Report format). The

student felt that Degree Navigator-UBC Report has a concise format, but then he felt there is

too much information in the requirement section and too little information in course

suggestion/selection. Also, the student felt that Degree Navigator is lack of diagrams and user

interactions. When asking the student about forming a plan after using the Degree Navigator,

the student claimed he will use the tool only for verification purpose, and he already developed

his own format in tracking his academic progress.

The student found the Degree Tracker looked as just as fine and concise as Degree Navigator,

but he felt that Degree Tracker gave him better understandings or intuitions in forming a plan

for his upcoming academic terms. He suggested adding course names beside course IDs and

placing the unsatisfied requirements before the completed requirements (he thinks unsatisfied

requirements are more important).

Subject 5

Subject Profile: The student is a third-year Computer Engineering student who just returned

from a year of COOP work terms.

Summary of Observations and Interview Feedback: The student just returned to UBC after

completed a COOP work term, and he often uses Degree Navigator 3 times per academic year

in his previous years at UBC. As a result, he claimed that he’s less familiar with the tool now.

The student chose the Full Progress format in Degree Navigator during the interview, and he

was frustrated by the massive information been presented to him. He mentioned that he only

look at the top 5% of the Full Progress format’s contents. The student found some

requirements are lacking comprehensive definitions which make him difficult to find the valid

courses to satisfy them. The student also has some unused credits that are actually valid for

some free electives. The student claimed that he will still go to the ECE website for assistance or

more helpful information after using Degree Navigator.

The student felt the Degree Tracker’s interface had shown enough information that’s relevant

to his program progress. The student still can’t understand some requirement names or the

requirement definitions in the Degree Tracker, however, he’s able to find some correct courses

to satisfy all the unsatisfied requirements presented in the tool. The student also suggested

putting course names beside the course IDs and adding a legend for defining the colours in the

progress bar.

Subject 6

Subject Profile: The participant is a recently-graduated student with a Electrical Engineering

degree and he is returning to UBC for his graduate study.

Summary of Observations and Interview Feedback: When the participant was studying at UBC,

he used Degree Navigator one to two times per academic terms. The student picked the UBC

Report format in Degree Navigator and found the webpage has simple presentation, but loaded

with too much information. The participant did not completed the section 3 tasks (Task 10 &

Task 11) successfully, and he claimed he would go to the ECE website for more information to

help solve problems related to course requirements and academic progress. The participant

also said he used Degree Navigator in the past just for checking compatibility between the

selected courses and the unsatisfied requirements, and not for planning his academic progress.

The participant felt fine with the Degree Tracker’s format and he was able to complete most

tasks successfully. Also, he was able to form a simple plan based on the information and

unsatisfied requirements presented on Degree Tracker. The participant also suggested adding

course names beside course IDs.

Subject 7

Subject Profile: The student is a third-year Computer Engineering student who just returned

from his military service.

Summary of Observations and Interview Feedback: The students uses Degree Navigator one to

two times per an academic term and he chose Full Progress format for this interview. The

student found the format looks fine overall, but only complains about too many extra

requirement information that are not applicable to his program. The student also could not

complete task 10 and task 11 successfully, and he mentioned that he would go to ECE website

for assistants in solving issues related to course requirements. The student also could not

generate a plan and claimed he needed to figure what course he can take to satisfy the

requirements first.

The student also liked the format of Degree Tracker, and he claimed this tool made him much

more easier to visualize what courses are required to complete the program. The student was

able to complete all the required tasks successfully and felt motivated in using the tool. The

student suggested putting grades and more definitions for some specific electives on the

Degree Tracker’s webpage.

Appendix D: List of References

[1] "Design Engineer Salary (Canada)", Payscale.com, 2018. [Online]. Available:

https://www.payscale.com/research/CA/Job=Design_Engineer/Salary. [Accessed: 24- Feb- 2018].