Comparative Evaluation: Anime Streaming Websites ......Sarah Letham (818) 667-5316 [email protected] age 2 Executive Summary Method: A comparative evaluation was performed

Comparative Evaluation: Anime Streaming Websites

Crunchyroll and Funimation

By

Sarah Letham

in Partial Fulfillment of the Requirements for the Degree of

Masters of Science in Human Factors in Information Design

at

Bentley University

For course HF751

Instructor: Bill Albert

Sarah Letham (818) 667-5316 [email protected]

Pag

e2

Executive Summary

Method:

A comparative evaluation was performed between anime streaming websites Crunchyroll and Funimation.

The two primary areas of interest were the usability of the video streaming service navigation and the usability

of account signup. Users performed six tasks, four of which were oriented towards the video streaming

navigation, and two of which were oriented towards account signup:

1. Find the latest episode of a show that is on the front page which can be viewed completely for free.

2. Navigate to the full episode list of the front page show.

3. Go to the first episode of a show that is not on the front page.

4. Navigate to second episode of the previous task’s show’s first episode page.

5. Go to where you would sign up for an account.

6. Where would you go to find out what paid account benefits are?

Users were asked a variety of demographic questions to determine their streaming habits, anime viewing

habits, and general commitment to being an anime fan. The sample contained a mix of non-fans, novice fans,

moderate fans, and ardent fans, with a slight tendency towards lesser fan commitment.

Most users had experience with general streaming sites Netflix and Hulu, and 52% of participants watch anime via online streaming. A third of users use Crunchyroll, half of whom have an account, while about ten percent use Funimation. None had a Funimation account. Results:

Crunchyroll’s primary advantages are in episode navigation, facilitating faster task times, and overall

usability/ease of use as measured by SUS.

Crunchyroll is at a severe disadvantage in teaching users about account benefits, and has generally poor

performance indicating to users the latest episode that can be watched without an account.

Crunchyroll had good performance but still has minor disadvantages in regards to account signup and helping

users find shows that are not promoted on the front page.


Pag

e3

Background and Methods

This study is a comparative evaluation of anime streaming websites Crunchyroll.com and Funimation.com.

Crunchyroll and Funimation are industry leaders in providing streaming anime content, particularly in offering

subtitled versions of episodes on the same day they air in Japan, a service that sets them apart from

competitors like Netflix.

The purpose of the study was to compare the usability of the two streaming sites on key aspects of streaming

content and account creation. The two primary research questions were:

1. How usable is the navigation for the video streaming service on each site?

Video streaming services are the primary value proposition of the site, so it is important that these are

usable. I also sought to compare the usability of navigating the streaming of promoted, front-page

shows and the usability of other non-promoted library titles.

2. How usable is the account signup page for each site, and do users understand the benefits of the paid

account?

Paid accounts are a primary source of revenue on these sites, so it’s important that the account signup

be usable and that users understand what they are getting when they sign up. Difficulty signing up

leads to abandonment, and mismatches between expectations and actual services leads to

cancellations.

I also aimed to answer whether users found the sites easy or difficult to use overall. After each task, the user

rated the ease of use on a seven-point Likert scale. At the end of the study, the user was given a ten-question

SUS questionnaire, modified for ease of use to be all positive.

This study was a between subjects design in which participants completed six tasks on either the Crunchyroll

site or Funimation site. The six tasks were identical between the sites, and shown in random order:

1. Find the latest episode of a show that is on the front page which can be viewed completely for free.

This task was chosen because typically streaming of episodes on the same day they air in Japan is

offered only as a premium service. After a period of time (typically about a week) the episode

becomes available for free. I wanted to test and see if new potential users could find the latest

episode that they are able to watch without first signing up.

2. Navigate to the full episode list of the front page show.

This task was chosen for its relevance to any user who isn’t caught up to the latest episode of a

show displayed on the front page of the website. The ability to navigate to whatever episode the

viewer is currently caught up to is important to the viewing experience.

3. Go to the first episode of a show that is not on the front page.

The front page has an extremely limited selection of the streaming website’s library, so another

vital service for a streaming site is facilitating navigation to titles that aren’t on the front page.

4. Navigate to second episode of the previous task’s show’s first episode page.

Navigation between episodes is as important as navigation between shows. This task was chosen

to test inter-episode navigation.

5. Go to where you would sign up for an account and enter information. You do not need to submit any

information or actually sign up.


Pag

e4

This task is to test the findability of the signup page. Account signup is a necessary part of

conversion.

6. Where would you go to find out what paid account benefits are?

This task and its accompanying pre- and post- test questionnaire were utilized to test how well

Crunchyroll and Funimation communicate their respective account value propositions.

The front page show for Task 1 of each site was chosen for its likelihood of remaining on the front page

throughout the test. As such, long-running popular shows were chosen. For Crunchyroll, Naruto Shippuden

was used, while for Funimation, Fairy Tail was chosen. However, it was unanticipated that Funimation would

completely rotate its front page shows on a regular basis even with the long-running Fairy Tail airing. As a

result, the front page show for Funimation had to be changed three times, since it was more important for the

basic task structure to stay intact than for the show to stay consistent. Five users completed task 1 on

Funimation for Fairy Tail, five completed for Puchim@s, and one completed for Jormungand. This is also

included in the “Limitations” section later in the report.

The other title was chosen to be a classic that anime fans would likely be familiar with, and which also would

not likely be removed from the library during the test. Unlike the front page show for Funimation, neither of

the non-front-page shows had to be changed during the test. Crunchyroll’s program for task 3 and 4 was

Puella Magi Madoka Magica, and Funimation’s was Spice and Wolf.

The following metrics were used to evaluate the data:

Task success o Binary – success/failure o Analysis of failure types, such as aborting task vs incorrect completion

Time on task in seconds

General ease of use o 7-point Likert scale rating ease of use after each question o 10-question SUS at the end

Qualitative information about users’ knowledge and expectations about accounts/ the site, as well as anime habits

Qualitative analysis of how people navigate through site o Do they typically employ the search bar? Go through episode lists? Navigate through show

titles?

Participants were a collection of friends, family, classmates, and friends-of-friends. Recruiting was done

through e-mail and Facebook contacts. The subjects’ familiarity with anime ranged from complete novices to

hardcore fans. As an incentive, a raffle for small anime-related prizes was done.

Task success data was collected from 22 participants, 11 using Crunchyroll and 11 using Funimation. Due to a

technical error all other data was lost for one participant. Therefore, outside of task success, all data was

drawn from 21 participants.


Pag

e5

User Demographics:

The user group was a mix of non-fans of anime, novice fans, moderate fans, and a few serious fans.

Demographic questions were asked to determine baseline viewing habits of users, particularly their familiarity

with video streaming and with anime. They were first asked what streaming sites they used, including

Crunchyroll, Funimation, and a sample of competitor sites. Participants were then asked to state which of

these sites they had an account for.

Users were asked how often they watch anime, whether they stream anime, and how they watch anime most

often. Users were also asked how many titles they own. The answer tiers, chosen based on anecdotal data,

represent four groups: those on the sidelines who just buy the a few classics, moderate fans who own a small

collection of favorites, serious fans who have a collection but don’t buy a lot of titles, and hardcore purchasers

who regularly purchase new content.

The following is a breakdown of their anime and streaming habits:

Most users had experience with general streaming sites Netflix and Hulu, with slightly less than half using

Amazon Video. 33% of users use Crunchyroll and a little over half have an account. Ten percent of

participants use Funimation’s website but none have an account. 19% of users have no streaming site account.

Another question found that 52% of participants watch anime via online streaming (see below).

33

10

90

71

43

0

29

0

19

0

62

14

43

010

19

0

10

20

30

40

50

60

70

80

90

100

Per

cen

t o

f U

sers

Po

lled

Streaming Site

Streaming Sites - Usage and Accounts (in Percent)

Select all sites that you use

Select all sites you have an account for


Pag

e6

Participants were asked how often they watched anime and how many titles they own in order to establish

whether our sample consisted primarily of novice, moderate, and hardcore anime fans.

Responses to these questions indicated that the sample was mixed. Those who responded that they never

watch anime were counted as non-fans, those who watch almost never were counted as novice fans, those

who watch “sometimes” were counted as moderate fans, and those who watch “often” or “all the time” were

counted as hardcore fans. The sample had a good mix with a slight tendency towards lesser fan commitment.

52%43%

19%14% 14%

29%

0102030405060708090

100

Per

cen

t o

f U

ses

Po

lled

Responses to: "How do you typically watch anime?"

29%

19%

38%

10%5%

0

10

20

30

40

50

60

70

80

90

100

Never Almost Never

Sometimes Often All the Time

Per

cen

t o

f U

sers

Po

lled

Responses to: "How often do you watch anime?"


Pag

e7

Similar results were found when we asked users how many titles they owned. More than half owned fewer

than five titles, indicating that they are likely non-fans or novice fans. The remaining participants were divided

between those who owned more titles.

Findings

The following is a summary of all results:

Task success (binary success and failure)

o Overall, the two sites had similar task completion rates

o Compared to Funimation, Crunchyroll is at a disadvantage in two places: teaching users about paid account benefits, and helping users find shows that aren’t being promoted on the front page

o Performance was very good on both sites for account signup and finding full episode lists.

o Both sites performed poorly on task time for finding the latest free episode of a show promoted on the front page

Analysis of failure types

o Four of six Funimation tasks included timing out among non-successes, compared to one for Crunchyroll

o Only Crunchyroll tasks had non-successes caused by abandonment

o On both sites, the majority of non-successes are subject errors (false-positive successes)

Time on task in seconds

o In all four video streaming navigation tasks, subjects using Funimation required significantly longer times to complete tasks than those using Crunchyroll

o Funimation had the same success as Crunchyroll for Account Signup, but was significantly faster on this one task

Fewer than 557%

5-1015%

11-2014%

More than 20

14%

Responses to: "How many anime series titles do you own?"


Pag

e8

o Crunchyroll clearly has an advantage in facilitating task speed in the navigation of episodes, but Funimation’s performance is equal or superior on account signup and finding account information

Click paths and page views

o 100% of people on Funimation’s website clicked the front page “Join” or “Subscribe” button for Task 5 (Account Signup)

o Users in Task 6 followed a nearly identical ratio of paths between websites, which was facilitated by Funimation’s design but not Crunchyroll’s

Users’ knowledge and expectations about paid account benefits

o Crunchyroll’s most well-known post-task feature was High Definition Streaming (70%)

o Increased Mobile Access jumped from 10% to 40% (the largest increase) between pre and post test

o Knowledge of store discounts increased from zero responses to 20%

General ease of use, as measured by a 7-point Likert scale rating ease of use after each question and a

10-question SUS questionnaire given at the end of the test

o Crunchyroll’s SUS score is 64.75

o This score places Crunchyroll at just below the 50th percentile for SUS

o Crunchyroll’s SUS score is much higher than Funimation’s score, which is just under 25th percentile

o Funimation’s account signup had the highest individual task rating

Task Success:

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Find the Latest Free Episode -

Front

Find Full Episode List -

Front

Find Episode 1 -Non-Front Page

Find Episode 2 from Episode 1

Account Signup Find Account Benefits Info

Rate of Task Success in Percent(Bars represent 90% Confidence Interval)

Crunchyroll Funimation

Per

cen

t o

f U

sers

wh

o S

ucc

essf

ully

Co

mp

lete

d T

ask


Pag

e9

Overall results of the comparison show that the two sites had similar task completion rates. Five of the six

tasks did not have significantly different results, with alpha at 0.1. However, two tasks had nearly significant

results, with p values under 0.2.

Task 6 (Find Account Benefits Information) was the only task with a truly significant difference, p<0.1. One

hundred percent of users were able to locate information about account details on Funimation, while only 40%

of Crunchyroll users were able to locate this information. This is a large gulf, one which Crunchyroll should be

concerned with. Crunchyroll is at a disadvantage when it comes to informing users about paid account

benefits.

While the results are not significant at the p=0.1 level, these results do suggest that Crunchyroll may be at a

disadvantage in facilitating finding episodes of shows in the library that aren’t being promoted on the front

page. For Task 3 (Navigate to Episode 1 of a non-front-page show), 91% of Funimation users successfully

located the first episode of the target program, while only 60% of Crunchyroll users could locate the episode,

p=0.13.

On Task 4 (Navigate to Episode 2 from Episode 1 of a non-front-page show) however, 73% of Crunchyroll users

successfully navigated to Episode 2 from Episode 1, while only 45% of Funimation users were able to do so,

p=0.19. Again, these results are not statistically significant at an alpha of 0.1, but may be significant with a

larger sample size.

Task 1 (Find the Latest Free Episode of a Front Page Show), Task 2 (Find the Full Episode List of a Front Page

Show), and Task 5 (Account Signup) had very similar performance between the two sites. The test found no

difference in the subjects’ ability to sign up for a new account, find the full episode catalog of a promoted

show, or find the latest free episode of a promoted show. In addition, performance was very good for Task 2

and Task 5, indicating that these tasks represent strengths of both sites. However, both sites performed

poorly on Task 1, indicating that users have a hard time on both sites with finding the latest free episode of a

show promoted on the front page.


Pag

e10

Failure Types:

The superior task success rate of Funimation that was described above does not tell the whole story. It is clear

from the chart that Crunchyroll has many more instances of non-successes caused by “Error,” indicating that

the user thought they were successful even though they had not actually completed the task.

Crunchyroll did not have any non-successes due to timing out with the exception of Task 4 (Navigate to

Episode 2 from Episode 1 of a non-front-page show). Meanwhile, four out of six Funimation tasks included

timing out among the non-successes, typically at an equal or higher rate than Crunchyroll’s Task 4. However,

only Crunchyroll tasks had non-successes caused by abandonment.

Statistically significant differences were found for Task 1 and Task 6 when comparing the makeup of failure

types between Funimation and Crunchyroll (Task 1 p=.04, Task 6 p=.03). The other tasks had more similar

rates of non-success causes, especially “Error” non-successes, which made up the overall majority of non-

successes. There was a clear effect overall of non-success type (p<0.01).

When compared with the time on task data in the next section, participant performance paints a picture of

Crunchyroll seeming easier than it is or difficult enough to cause abandonment, while Funimation’s tasks took

a longer time (causing many more timeouts) but no abandonments and fewer false impressions of success. It

is not clear if there is an element to Funimation’s design that encourages users to attempt their task for a

longer period of time; this is a potential element to explore in the future.

60%44%

100%

67% 75%

100%

33%

71%

100% 100% 100%

40%25%

33%

56%33% 33% 29%

100%

Type of Non-Success by TaskError Abandon Timeout


Pag

e11

Time on Task:

Significant differences were found in mean task time between Crunchyroll and Funimation for Task 1 (Find the

Latest Free Episode of a Front Page Show) (p<0.01), Task 2 (Find the Full Episode List of a Front Page Show)

(p=0.05), Task 3 (Navigate to Episode 1 of a non-front-page show) (p=0.01), and Task 4 (Navigate to Episode 2

from Episode 1 of a non-front-page show) (p=0.02). The p value for the difference found in Task 5 (Account

Signup) was only 0.11, nearly equal to the target value 0.1, and so should be essentially considered significant

as well.

Time data was cleaned up using the method of discarding as outliers any scores more than two standard

deviations away from the mean.

In tasks 1-4, Funimation’s design required significantly longer task times to complete than Crunchyroll’s design.

While Funimation scored better in the task success section, Funimation only performed significantly better

than Crunchyroll on Task 6, for which there is no difference in time on task. Funimation had the same success

as Crunchyroll for Account Signup, but was significantly faster. Crunchyroll clearly has an advantage in

facilitating task speed in the navigation of episodes, though this advantage does not extend to account signup

and finding account information.

Comparing scores on tasks within Crunchyroll, significant differences were found between many of the tasks.

Task 2 had the fastest mean speed at 29 seconds, significantly faster than Task 1, Task 3, and Task 6 (p<0.1).

Task 1 and Task 6 had the longest times, averaging 72 seconds.

0

20

40

60

80

100

120

140

160

180

Task

Tim

e in

Sec

on

ds

Mean Task Time (in seconds)


Pag

e12

These data support the earlier suggestions that Crunchyroll should focus its energies on helping users navigate

front page episodes and increasing access to information about account benefits.

Click Paths and Page Views:

Videos of participant performance were examined to find out where users go first in trying to navigate their

task. The majority of users clicked on a link somewhere on the front page (typically above the fold) such as

“Shows” or an episode title. The next most common option was to utilize the search bar. The other options

were divided between a catch-all Other category (used to denote clicking on, for example, DVD ads or

navigating out of the site, or going down a path that did not fit into any other category) and between links

from the bottom navigation bar.

It is noteworthy that 100% of people on Funimation’s website clicked the front page “Join” or “Subscribe”

button for Task 5 (account signup), which likely contributed to its extremely low task time and high success

rate.

It is also noteworthy that users in Task 6 followed a nearly identical ratio of paths between websites. Since

Funimation had a significantly higher task success rate on this task, it is clear that Funimation’s design is

congruent with the task paths chosen by users, while Crunchyroll’s design is not. Crunchyroll should update its

design so that paths to account information are easier to find with intuitive paths.

38%

82%

50%73%

29%

82%

50% 45%63%

100%

75% 73%

38%

18%

38%

27%

43%

18%

9%

13%

25% 27%25%13%

29%50% 45%

25%

0%10%20%30%40%50%60%70%80%90%

100%

Per

cen

t o

f U

sers

First Click in Task by Percent of Users

Front Page Link* Search Bar Bottom Bar Other


Pag

e13

The average number of unique page views and average number of clicks were examined to find differences

between the sites, especially to be compared with time on task. However, few significant or differences were

found among these data. The lowest scores, on Crunchyroll for finding the full episode list of a front page

show and finding the first episode of a non-front page show, coincide perfectly with the highest proportions of

using the search bar on Crunchyroll. Use of the search bar on Funimation did not decrease the number of

clicks nearly as much, which is unsurprising considering that Funimation’s search function involves navigating

to a separate search page rather than using a drop-down selection like on Crunchyroll.

Account Benefit Understanding:

Task 6 was used to determine what users knew about paid account benefits. For each site, users were

presented with a pre-test and post-test. The list given was the same for both shows.

Crunchyroll’s questionnaire contained six actual account benefits (No Commercials, High Definition Streaming,

See Episodes Earlier, Store Discounts, Access to more shows/titles, Increased Mobile Access) and four false

account benefits (Watch Dubbed Episodes, Watch Uncut Mature Shows, Episode Upload Notification, Episode

Downloads). Ten users were asked to identify the paid account benefits of Crunchyroll.

Funimation’s questionnaire contained eight actual account benefits and two false account benefits. Actual

account benefits were No Commercials, High Definition Streaming, See Episodes Earlier, Store Discounts,

Access to more shows/titles, Increased Mobile Access, Watch Dubbed Episodes, and Watch Uncut Mature

Shows. Episode Upload Notification and Episode Downloads are still false account benefits for Funimation.

Eleven users were asked to identify the paid account benefits of Funimation.

02468

1012141618

Mea

n V

iew

sMean Unique Page Views and Clicks(Bars represent 90% Confidence Interval)

Mean Unique Pageviews Mean Number of Clicks


Pag

e14

Crunchyroll:

Crunchyroll Pre-Test results

In the pre-test before finding the information page, half of users identified high definition streaming and

seeing episodes earlier as account benefits. 40% of users answered that no commercials and greater access to

shows were account benefits. The “Other” category results consisted of comments that the user was

unfamiliar with the site and so did not know the answers.

One user mentioned increased mobile access. No users were aware that Crunchyroll offers store discounts as

an account benefit.

Episode Downloads was the most common false positive, with 30% of users identifying this service as an

account feature. Episode upload notification and watching dubbed episodes were picked by one participant

each.

Crunchyroll Post-Test results


Pag

e15

Performance was overall better on the post-test questionnaire, which users took after completing the task. It

is noteworthy that only 40% of users successfully completed task 6, so it is unsurprising that many users’

accuracy did not improve.

The most well-known feature after the post-test questionnaire was High Definition Streaming, which was

identified by 70% of the participants, followed by Access to More Shows/Titles, which was identified by 60%.

No Commercials increased from 40% to 50%, and Increased Mobile Access jumped from 10% to 40% (the

largest increase). Customers learned about store discounts, increasing from zero responses to 20%.

False positives also increased, indicating that the results are not entirely due to the information from the task.

Watch Uncut Mature Shows increased from 0% to 10%, Episode Upload Notification increased from 10% to

20%. However, Episode Downloads did decrease from 30% to 20%, and Watch Dubbed Episodes remained at

10%, so the increase was not across the board.

Oddly, the actual account feature See Episodes Earlier decreased in response rate from 50% to 40%, but this

may be a result of the term used. In the “Other” comments one user mentioned Simulcast as another benefit,

which was intended by the researcher to be covered by “See episodes earlier.” Had the word “simulcast” been

used, it is possible that more users would have recognized the feature.

Additional answers in the “Other” category indicate that users did not have enough time to read and commit

to memory the necessary information for the questionnaire. This is discussed further in the “Limitations”

section below.

Funimation:

Funimation Pre-Test Results

The feature with the highest response rate was No Commercials, with 64%. This is followed by Access to More

Shows/Titles, with 45% response rate. Third place is tied at 27% between See Episodes Earlier, Store Discounts,

and the false positive answer Episode Upload Notification.


Pag

e16

Only 18% of users identified High Definition Streaming as an account benefit, the same number as thoe who

erroneously identified Episode Downloads as a feature. Only 9% of users identified Watch Dubbed Episodes,

Watch Uncut Mature Shows, and Increased Mobile Access.

Funimation Post-Test Results

Overall performance was improved by completing Task 6, even more than for Crunchyroll. No Commercials

held its place with 64% response rate, while Access to More Shows/Titles increased from 45% to 55%. High

Definition Streaming had the greatest increase, going from 18% to 45% response rate. See Episodes Earlier

also increased, from 27% to 36%. Watch Dubbed Episodes, Watch Uncut Mature Shows, and Increased Mobile

Access all increased from 9% to 18%.

The only account feature for which response rate decreased was Store Discounts, which decreased from 27%

to 9%.

The false positives for Episode Upload Navigation decreased from 27% to zero. Episode Downloads stayed the

same, with 18% response rate. There was no increase in false positives.

While improvement differed between Funimation and Crunchyroll, aspects of the task implementation may

have influenced the results in such a way that prevents drawing conclusions from these differences. The

Limitations section discusses these possible influences in further detail.

Post Task Ratings and Overall SUS:

At the conclusion of the tasks, participants were presented with a ten question SUS questionnaire measuring

overall satisfaction with the site. A SUS questionnaire with all positive wording was used in order to prevent

confusion on the part of the participant.


Pag

e17

The SUS score for Crunchyroll, found by averaging scores for each participant, was 64.75. This score places

Crunchyroll at just below the 50th percentile for SUS, indicating below average performance. It should be

noted that if one possible outlier score of 15 is removed, the mean score increases to 70.28, which is a little

over 50th percentile.

The SUS score for Funimation was 54.77. This score places Funimation at just below the 25th percentile for SUS,

indicating poor performance. If one possible outlier score of 12.5 is removed, the mean score increases to 59,

which is over the 25th percentile but still far under the 50th.

Crunchyroll’s SUS score is quite a bit higher than Funimation’s score, but Crunchyroll should still aim to

improve its SUS score so that customer satisfaction better compares with other websites.

Post-task ratings measured on a seven-point Likert scale were also compared. Satisfaction ratings were

measured for those who were successful, those who were unsuccessful, and then the two ratings were

combined:

Unsurprisingly, Funimation’s account signup had the highest rating. Interestingly, successful users rated Task 2

on Crunchyroll very highly, but the 9% of unsuccessful users gave Task 2 the lowest rating, zero. This is not a

major concern since 91% of users were successful on this task and they rated the task as very easy, but it is

worth noting cases like Task 2 where unsuccessful users found the task very hard and successful participants

found it very easy. In most other cases, ratings were similar across the board.

0

1

2

3

4

5

6

Mea

n E

ase

of

Use

Rat

ing

(0-6

)

Ease of Use Rating by Task and Performance

Mean Ease Rating (Successful)

Mean Ease Rating (Unsuccessful)

Mean Ease Rating (All)


Pag

e18

Limitations and Possible Extensions

Due to changes in the Funimation front page, the title chosen for the first and second task on

Funimation had to be rotated during the test in order to keep the task consistent. The following shows

were used:

o 1 complete: Jormungand

o 5 completes: Puchim@s

o 5 completes: Fairy Tail

Future studies could avert this by timing the study to have one show throughout, changing the task so that it

did not involve the front page show, or working directly with Funimation to make sure one show remains on

the site throughout the test.

Crunchyroll title Puella Magi Madoka Magica is more complex than Spice and Wolf, so it is easier to

utilize the search function for Funimation’s title than Crunchyroll’s title. However, the task with title

was visible at all times during the test, so it would be possible for users hoping to utilize the search

function to copy and paste Puella Magi Madoka Magica. It is still possible that the longer title was

more difficult to work with, so future studies should use titles of similar length, word familiarity, and

word complexity in order to eliminate this potential confounding factor.

The Task 6 questionnaire was identical for both shows, meaning that there were four false options for

Crunchyroll and two false options for Funimation. As such, the Funimation test had an unfair

advantage against false positives. As a result, the questionnaire performance for Task 6 was not

directly compared between sites.

Use of non-randomly chosen convenience samples was also a limitation, as was the limited number of

participants. Future studies should use a larger sample of randomly selected representative

participants.

For task 6, not enough time was left for users to read the page. In addition, it was not made clear

enough that users needed to review the information on the site. The reason is that for all other tasks,

immediate rerouting to the questionnaire is the sign that the task is complete, and I didn’t want

participants to navigate away from the correct page assuming it was incorrect. In future studies,

subjects should be given enough time to read the page, and they should be clearly instructed about

the learning aspect and that this task’s success notification would be different.

It is the researcher’s first time using User Zoom, so there was a certain amount of trial and error in the

setting up of the test and in the analysis of results. Some analysis of results was made more difficult

due to the setup of UserZoom; for example, the tasks were easily divided into two groups, but

questionnaire results were not. The details of the study plan, setup, and implementation might be

better planned and executed by a researcher more familiar with User Zoom.


Pag

e19

Besides task success results, data was lost for one participant in the Crunchyroll Group who did not

officially finish the final question in User Zoom. As a result, that user was not counted by UserZoom as

a completed participant, so the results were discarded.

Additional opportunities for future research exist in studying why users are more likely to spend time

attempting to finish Funimation tasks but more likely to abandon Crunchyroll tasks (or erroneously

mark them as successful).

Conclusions

Crunchyroll’s strongest areas are:

Episode navigation, including inter-episode navigation and finding full episode lists

Facilitating faster task times across tasks, particularly those involving navigation

Overall usability and ease of use as rated by SUS when compared to Funimation

Crunchyroll’s weakest areas are:

Informing users about account benefits o Users in Task 6 followed a nearly identical navigation strategy regardless of website. This

navigation strategy was facilitated by Funimation’s design but not Crunchyroll’s. o Users did not know at all about store discounts being a benefit of Crunchyroll’s paid account, and

on the Task 6 post-task questionnaire the responses increased only to 20%. These scores fell below the otherwise typical 40-70% range of users knowing about the account benefits.

Helping users identify the most recent episode that can be watched without an account o It is not clear whether this is due to users not being able to distinguish what episodes are

restricted to premium viewers, due to being unable to find the latest episodes in general, or if there is another cause. Further investigation is warranted.

Other areas to improve:

These are strengths of the competition where Crunchyroll is relatively less effective:

Account signup o 100% of people on Funimation’s website clicked the front page “Join” or “Subscribe” button for

Account signup. For comparison, 63% of users clicked a front page direct link to sign up for an account.

Facilitating navigation to shows that are not promoted on the front page o Recall that 91% of Funimation users successfully located the first episode of the non-front-page

target program, while only 60% of Crunchyroll users could locate the episode, p=0.13

Additional Recommendations for Crunchyroll:

Try to further improve usability and ease of use in order to better compare to other websites.

Crunchyroll’s score is higher than Funimation’s, but is still lower than 50% of the websites tested.


Pag

e20

Crunchyroll should aim to not only beat the SUS scores of their direct competition, but even peripheral

competitors such as general streaming sites like Netflix. If Crunchyroll can aim high enough to take its

navigation to the next level and provide a seamless user experience, the result should be increases in both

conversion and positive word of mouth.

Look into why users were more likely to abandon tasks on Crunchyroll or erroneously mark them

successful, but more likely to keep at a task for a longer period of time on Funimation. The investigation

into this difference may lead to other fruitful avenues for improving Crunchyroll’s usability.

Documents

Comparative Evaluation: Anime Streaming Websites ......Sarah Letham (818) 667-5316 [email protected] age 2 Executive Summary Method: A comparative evaluation was performed