YouTubeGuidelines.pdf

8/15/2019 YouTubeGuidelines.pdf

1/64

YouTube Rating Guidelines

Version 1.1

January 8, 2016

1. Introduction2. Important Notes3. Types of YouTube Search Results

3.1. Video Results3.2. Channel Results3.3. Playlist Results3.4. Movie Results3.5. Pay-Restricted Results

4. Assigning Utility Rating

4.1. Utility Rating for Video Results4.1.1. Video Result Utility Rating Examples

4.2. Utility Rating for Channel Results4.2.1. Channel Result Utility Rating Examples

4.3. Utility Rating for Playlist Results4.3.1. Playlist Result Utility Rating Examples

4.4. Utility Rating for Movie Results4.4.1. Movie Result Utility Rating Examples

4.5. Utility Rating for Pay-Restricted Results4.5.1. Pay-Restricted Result Utility Rating Examples

5. Assigning Quality Rating5.1. Quality Rating for Video Results

5.1.1. Video Result Quality Rating Examples

5.2. Quality Rating for Playlist Results5.2.1. Playlist Result Quality Rating Examples

5.3. Quality Rating for Channel Results5.3.1. Channel Result Quality Rating Examples

5.4. Quality Rating for Pay-Restricted Movies and Videos5.4.1. Channel Result Quality Rating Examples

6. Assigning Flags to YouTube Result Blocks6.1. Assigning flags to video results6.2. Assigning flags to non-video results

7. Reporting Results with Duplicate Landing Pages7.1. Pre-Identified Duplicates7.2. Rater-Identified Duplicates7.3. Reporting Duplicate Results

8. Assigning an Overall Side by Side Score (if requested)

1

Proprietary and Confidential - 2016


2/64

1. Introduction

The goal of YouTube search is to provide users with relevant, engaging videos. The data derived from your rating

tasks is used to inform YouTube search quality engineers about what can be done to achieve this goal. These

guidelines are designed to help make your responses on rating tasks as useful as possible.

In YouTube Block Utility tasks, you will be given a query that was issued on www.youtube.com and one or two sets of

YouTube result blocks to evaluate.

In all YouTube Block Utility tasks, you will:

1. Assign a Utility Rating to each YouTube result block using the given Utility Rating slider.

2. Assign a Quality Rating to each YouTube result block using the given Quality Rating slider.

3. Assign Flags to YouTube result blocks, when appropriate.

4. Report Duplicate Result Blocks (results that have the same landing page).

In some YouTube Block Utility tasks, you will also be asked to assign an Overall Side by Side rating. You will be

required to leave a comment if you think one side is better than the other side.

These guidelines provide information about the different types of results provided by YouTube and how to evaluate

them. Please note that YouTube continually makes changes to the appearance of pages on the site, so some of the

screenshots in these guidelines may not exactly match what you see on the site.

2. Important Notes

Result blocks are sometimes slow to load: Before assigning any ratings, please scan the task page to make sure

that all result blocks have loaded. Please note that result blocks may be slow to load and/or that the task page may

need reloading if there are empty result blocks. If the result blocks fail to load after refreshing a few times, please release the task and send a note to your employer.

Sometimes a side will not have any result blocks : If one side displays the message “This side did not generate

any results”, please do not release the task or send a note to your employer. Please assign utility ratings to each

result block using your best judgment. Please also assign an overall SxS rating, if the task asks for one.

Sometimes the task will not be divided into two sides: You will be given just one set of result blocks to evaluate.

Do not refresh the task page after assigning ratings: If you refresh the page after assigning ratings, you will lose

them and they will have to be entered again.

2


u=hkpjohpj

http://www.youtube.com/


3/64

3. Types of YouTube Search Results

Most YouTube search results are videos, channels, or playlists. These guidelines provide guidance on rating these

types of results.

Result Type Examples

Videohttps://www.youtube.com/watch?v=uBWrpVrazzA

http://www.youtube.com/watch?v=W795W63n7mA

Channel

https://www.youtube.com/user/PewDiePie

http://www.youtube.com/user/whitehouse

https://www.youtube.com/channel/UCE80FOXpJydkkMo-BYoJdEg (topic channel)

Playlist

http://www.youtube.com/watch?v=e3i6LSW_Kyg&list=UUYxRlFDqcWM4y7FfpiAN3KQ

http://www.youtube.com/watch?v=JGPQcGu2Cso&list=UUGm3CO6LPcN-Y7HIuyE0Rew

https://www.youtube.com/watch?v=EVBsypHzF3U&list=RDEVBsypHzF3U (mix)

3


http://www.youtube.com/watch?v=qr5VqBP1voA&list=FL81VD6eeuLLSfyY_D-N8sVwhttps://www.youtube.com/watch?v=EVBsypHzF3U&list=RDEVBsypHzF3Uhttp://www.youtube.com/watch?v=JGPQcGu2Cso&list=UUGm3CO6LPcN-Y7HIuyE0Rewhttp://www.youtube.com/watch?v=JGPQcGu2Cso&list=UUGm3CO6LPcN-Y7HIuyE0Rewhttp://www.youtube.com/watch?v=JGPQcGu2Cso&list=UUGm3CO6LPcN-Y7HIuyE0Rewhttp://www.youtube.com/watch?v=e3i6LSW_Kyg&list=UUYxRlFDqcWM4y7FfpiAN3KQhttp://www.youtube.com/watch?v=e3i6LSW_Kyg&list=UUYxRlFDqcWM4y7FfpiAN3KQhttp://www.youtube.com/watch?v=e3i6LSW_Kyg&list=UUYxRlFDqcWM4y7FfpiAN3KQhttp://www.youtube.com/user/whitehouse?https://www.youtube.com/channel/UCE80FOXpJydkkMo-BYoJdEghttp://www.youtube.com/user/whitehousehttp://www.youtube.com/user/madonna?http://www.youtube.com/user/madonna?https://www.youtube.com/user/PewDiePiehttp://www.youtube.com/watch?v=W795W63n7mAhttps://www.youtube.com/watch?v=uBWrpVrazzA


4/64

This is how YouTube result blocks for these result types were displayed at the time these guidelines were written:

Result Type Result Block

Video Result

Channel Result

Notice the “CHANNEL”label

Playlist Result

Notice the thumbnaildifference. Usercurated playlist resultand YouTubeauto-generated mixplaylist result could lookdifferently.

YouTube provides other types of search results, such as movie results, artist watch cards, and show watch cards, and

you may occasionally be asked to rate these other result types. Please use your judgment and assign reasonable

ratings based on the utility of the landing page to the query.

This is how YouTube result blocks for these result types were displayed at the time these guidelines were written:

Result Type Result Block

Movie Result

4



5/64

Artist Watch Card

Show Watch Card

5



6/64

3.1. Video Results

Video results are the most common type of YouTube search result. Here is a link to the video that is displayed when

the user clicks on the title 10 Cutest Cat Moments of this video search result:

The video result page looks like this at the time these guidelines were written:

6


http://www.youtube.com/watch?v=q1dpQKntj_w


7/64

3.2. Channel Results

A channel is a collection of YouTube webpages. Here are some signs that usually indicate that a particular YouTube

webpage is part of a channel:

● A channel specific banner at the top the page, followed by the channel title, a “Subscribe” button, and the number of subscribers.

● Navigational tabs, such as “Home”, “Videos”, “Popular”, “Channels”, “Discussions”, “About”, etc.

● Prominent display of the word “channel” on the page.

● A search box titled “Search Channel”.

All channels have at least one of these signs. With a little experience you will soon be able to recognize channels at a

glance.

Here are some examples. Please visit these channel pages by clicking the screenshots or the titles above the

screenshots.

Channel Example 1: Taylor Swift Channel Channel Example 2: Kitesurf Channel

Examples 3 and 4 below are known as Topic channels. Topic channels are created automatically by YouTube, and

the topic channel names usually start with a hashtag (#).

7


http://www.youtube.com/user/puehsetennis?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/ladygagavevo?https://www.youtube.com/user/TaylorSwiftVEVO


8/64

Example 3: Taylor Swift Topic Channel Example 4: Minecraft Topic Channel

When you click on the about tab, you’ll see topic channels are auto-generated by YouTube.

8


http://www.youtube.com/channel/HCfvBjoGqrZq0http://www.youtube.com/channel/HCfvBjoGqrZq0http://www.youtube.com/channel/HCfvBjoGqrZq0http://www.youtube.com/channel/HCfvBjoGqrZq0http://www.youtube.com/channel/HCfvBjoGqrZq0http://www.youtube.com/channel/HCfvBjoGqrZq0https://www.youtube.com/channel/UCQvWX73GQygcwXOTSf_VDVghttp://www.youtube.com/channel/HCQrFlYXe6zIIhttps://www.youtube.com/channel/UCPC0L1d253x-KuMNwa05TpA


9/64

9



10/64

3.3. Playlist Results

A playlist is a list of videos arranged in a linear order. The arrangement of the videos is controlled by the registered

YouTube user who owns and creates the playlist, or by the automatic mix algorithm from YouTube. Under the default

playlist settings, videos in the playlist will play sequentially with no further action from users. Users may click videos in

the playlist if they wish to watch videos in a different order.

This is how YouTube Playlist result blocks were displayed at the time these guidelines were written:

The playlist result page looks similar to a video page, with an additional list of videos on the right side above the

related videos panel.

10



11/64

11



12/64

3.4. Movie Results

Some movies on YouTube are free for users. Others require payment and users would not be able to view them

without paying. When assigning a utility rating to such movies, please assume that the movie plays as advertised. If

the title of the movie is a good fit for the query, it should get a good utility rating. You will learn more about the YouTube video rating scale in Section 4.

Here are examples of free movies and movies that require payment.

Movie Result

TypeResult Block

Free Movie Result

Paid Movie Result

Notice the dollar signblue label and the“$4.99” fee

12



13/64

3.5. Pay-Restricted Results

Pay-restricted results are YouTube search results that cannot be viewed without first making a payment. These

results have a price icon somewhere in the result block which looks like this:

Some of the movie and video results are pay-restricted. Upon clicking on a pay-restricted result, the preview trailer

might start to play. Please do not confuse such preview trailers with actual movies!

Sometimes, the video result directs to a YouTube page with a video player informing users that payment is required

before the video can be viewed.

13



14/64

14



15/64

4. Assigning Utility Rating

Understanding the query and user intent

As with other types of rating, you should familiarize yourself with the possible interpretations of the query and develop an understanding of the user’s intent. If the meaning of the query is not perfectly clear, do some research: look at the

YouTube search results in the task and/or try issuing the query on google.com or wikipedia.com. You may also be

able to find definitions of unfamiliar terms at urbandictionary.com. Try to imagine the various reasons a user might

have issued the query on YouTube and the kind of videos that the user expects to see.

Evaluating YouTube Video Result Blocks

The same rating scale will be used when evaluating all types of YouTube results; however, you will be given individual

instructions for rating these four types: videos, channels, playlists, and movies.

4.1. Utility Rating for Video Results

The usefulness of a YouTube video result is determined by the video result (the main video on the landing page) and

by the user’s intent. If the user has a particular result (video) in mind, then that result is very useful. Results that have

nothing to do with the user’s intent are not relevant and therefore not useful.

You will assign a utility rating to YouTube video results according to the following scale:

Rating Description

VeryUseful

Use this rating when the video is not only clearly relevant to the query, but is a video that a user typing the querywould expect to see and be pleased to see.

UsefulUse this rating when the video has a clear connection to the query, and is a result that some users would be

pleased to see but would not appeal to as wide an audience as a Very Useful

video.

MarginallyUseful

Use this rating when the video has a connection to the query, but it would not appeal to the vast majority of userswho issue the query.

This rating is also appropriate for video results that serve a minor interpretation of the query.

Useless

Use this rating:

● When the video has no clear connection to the query and would evoke the response “I have no ideawhy they are showing me this!” in the vast majority of users.

● The video does not play. You will also check the Did Not Load flag.● The video is unratable because it is in a foreign language and you are unable to determine utility. You

will also check the Foreign Language flag.

This rating also applies when you cannot determine the utility of the video within the allotted time.

IMPORTANT: Do NOT use this rating when the video requires payment to play. Some movie results requirepayment to play. For movies that do not play without payment, please assume that the movie is what it claims tobe.

Important: You are not required to assign the Useless rating to foreign language videos. If you believe that a foreign

language video may have some utility, you may select another rating. Here are two possible scenarios:

15


http://urbandictionary.com/http://wikipedia.com/http://google.com/


16/64

● The video doesn’t need language to be appreciated. For the query [skateboarding], English (US), this Chinese video may have some utility for English users.

● The user is looking for a foreign language video. For the query [Como la Flor by Selena], English (US)

this Spanish video is exactly what the user is looking for. In fact, this video is Very Useful even though it is in

a foreign language.

4.1.1. Video Result Utility Rating Examples

Query URL Rating Description

[baby pandasneeze]

http://www.youtube.com/watch?v=FzRH3iTQPrk

Very UsefulVideo page with a video of a baby pandasneezing

[zidane headbutt] http://www.youtube.com/watch?v=zAjWi663kXc Very Useful

Video page with a video of Zinedine Zidane

giving a headbutt to an Italian soccer playerduring the 2006 FIFA World Cup match.

[pug] http://www.youtube.com/watch?v=Njo28CN5XuY

UsefulVideo page with a video of a talking pug on theMontel Williams TV Show.

[elf] http://www.youtube.com/watch?v=9jyCfRHumHU

UsefulVideo page with a video clip of the movie “TheElf”.

[snakes on a plane] http://www.youtube.com/watch?v=J2IZkwgXl8M

Marginally UsefulVideo page with a video of the University ofSouthern California Marching Band playing asong from the movie “Snakes on a Plane”.

[Star Wars] http://www.youtube.com/watch?v=hVrIyEu6h_E

Marginally UsefulVideo page with a video of “Grocery StoreWars” with animated fruits and vegetablesplaying characters from the movie, “Star Wars”

[hot dog] https://www.youtube.com/watch?v=lzQ8GDBA8Is

UselessVideo page with a video with a recording ofElvis Presley singing “Hound Dog”.

[oscar ceremony] https://www.youtube.com/watch?v=Avp-KwMmZA8

UselessVideo page with a video of a Japaneseanimation with a main character named Oscar.

[modern family] http://www.youtube.com/watch?v=59twO1fJwtQ

Useless Video page with video from the TV show “TheView”, not “Modern Family”

16


http://www.youtube.com/watch?v=59twO1fJwtQhttp://www.youtube.com/watch?v=59twO1fJwtQhttp://www.youtube.com/watch?v=OU6EHnDAAvQhttps://www.youtube.com/watch?v=Avp-KwMmZA8https://www.youtube.com/watch?v=Avp-KwMmZA8http://www.youtube.com/watch?v=hVrIyEu6h_Ehttp://www.youtube.com/watch?v=hVrIyEu6h_Ehttp://www.youtube.com/watch?v=hVrIyEu6h_Ehttp://www.youtube.com/watch?v=J2IZkwgXl8Mhttp://www.youtube.com/watch?v=J2IZkwgXl8Mhttp://www.youtube.com/watch?v=J2IZkwgXl8Mhttp://www.youtube.com/watch?v=9jyCfRHumHUhttp://www.youtube.com/watch?v=9jyCfRHumHUhttp://www.youtube.com/watch?v=9jyCfRHumHUhttp://www.youtube.com/watch?v=Njo28CN5XuYhttp://www.youtube.com/watch?v=Njo28CN5XuYhttp://www.youtube.com/watch?v=zAjWi663kXchttp://www.youtube.com/watch?v=zAjWi663kXchttp://www.youtube.com/watch?v=zAjWi663kXchttp://www.youtube.com/watch?v=FzRH3iTQPrkhttp://www.youtube.com/watch?v=FzRH3iTQPrkhttps://www.google.com/evaluation/portal/portal_files/VideoResultRating.pdfhttps://www.google.com/evaluation/portal/portal_files/VideoResultRating.pdfhttp://www.youtube.com/watch?v=gXqwO9ZsPLchttp://www.youtube.com/watch?v=N-AdUM0Y0ychttp://www.youtube.com/watch?v=N-AdUM0Y0yc


17/64

4.2. Utility Rating for Channel Results

The first step in evaluating a channel result is to determine whether the query is channel-seeking (“navigational”) or

non-channel-seeking (“non-navigational”). In other words, you should try to determine whether the query is looking for

a particular channel. Note: the term “navigational” is used here in the context of YouTube channel results, not Web

results.

● Channel-seeking queries. We assume that the user is looking for a YouTube channel when:

a. The word “channel” or “official” is contained in the query. Here are some examples: [madonna

official], [taylor swift channel], [sesame street official channel], [elton john vevo]. Note: Vevo is a

music website, and channels of Vevo-affiliated artists are displayed on YouTube.

b. The query is an unusual word or name which is only associated with a registered YouTube channel

user. Here are some examples: [CJTClol], [kjgophish], [jimmykimmellive].

● Non-channel-seeking queries. We will assume that the user is not looking for a particular YouTube

channel result for all other queries.

Users who visit channel pages are typically interested in either going to a particular channel or finding videos on aparticular topic.

● A particular channel. A user may visit Lady Gaga’s channel to see if Lady Gaga has posted any recent

videos or to see if Lady Gaga or her fans have posted new messages. Similar content may be available on

other channels, but the user is interested in this particular channel.

● Videos on a particular topic. A user may visit a particular channel to find a good collection of videos on a

particular topic. The channel owner and the specific channel may not be important to the user – it is the topic

that is of interest. For example, a user interested in snowboarding may visit a channel devoted to

snowboarding to peruse the collection of videos on that channel.

It is important to understand these two cases because the rating you assign to channel results should depend on the

user intent.

You will assign a utility rating to channel results for channel-seeking queries according to the following scale:

Rating Description

Very UsefulUse this rating when you are certain that the user is looking for the particular channel. For example:[madonna channel]

UsefulUse this rating when you believe it is

likely or possible

but not certain that the user is looking for theparticular channel. For example, the query has multiple interpretations and this interpretation is likely orpossible, but not dominant.

Marginally Useful

Please do not use this rating when the query is channel-seeking. The channel will either be the target of

the query (

Very Useful

), one of several likely targets of the query (

Useful

), or the wrong channel( Useless).

Useless Use this rating when you believe the user is looking for a different channel.

For non-channel-seeking queries, please watch parts of the three videos on the channel that seem to be most

relevant to the query. Based on your impression of these three videos, please assign a rating according to the

following scale:

17



18/64

Rating Description

Very Useful Please do not use this rating when the query is non-channel-seeking.

Useful

Use this rating when the following conditions are true:

● The ratio of good-to-bad videos is very high. Very Useful and Useful videos are consideredgood videos. Marginally Useful and Useless videos are considered bad videos.

● Most of the best videos on the channel are easy to find on the channel landing page; that is, thepage that loads when you click the title of the channel result.

Marginally UsefulUse this rating if one or more of the above conditions is somewhat lacking. For example, the ratio ofgood to bad videos is even lower and/or many of the best videos are hard to find.

UselessUse this rating when most of the videos are bad and/or the good videos are difficult to find, i.e., the goodvideos are not found on the landing page. For example, if you have to go to another page or if you haveto use the search box on the channel page to find good videos, the good videos are difficult to find.

4.2.1. Channel Result Utility Rating Examples

QueryChannel-Seeking?

ResultTarget ofthe Query?

Rating Explanation

[rihannachannel]

YesRihanna’sChannel

Yes Very UsefulThis channel is the target of the query, soit is Very Useful .

[rihannachannel]

YesRihanna - Stayft. Mikky Ekko

No, not achannelresult

UsefulThis is a specific video from Rihanna'schannel, but since the query is channelseeking, the best video results are Useful .

[lada gaga vevo] YesLady Gaga’sVevo Channel

Yes Very Useful The channel is the target of the query, so itis Very Useful .

[lady gaga] NoLady Gaga’sVevo Channel

Not achannel-seeking query

Useful

This channel is Useful because it isrelated to the artist, the ratio ofgood-to-bad videos is high, and the goodvideos are easy to find.

[chris brownofficial]

YesMadonna’sChannel

No UselessEven though this is a high quality channel,it is not the channel the user is seeking.

[til it happens Toyou]

NoLady Gaga’sVevo Channel

Not achannel-see

king query

Useful

The user seeks a specific video which iseasily found (at this writing) on the channellanding page. Other videos on the channelare Useful

to the query.

[bad romance] NoLady Gaga’sVevo Channel


MarginallyUseful

Even though the channel has a high ratioof good-to-bad videos, “Bad Romance”(the video specified in the query) is not onthe landing page making it hard to find.This channel is therefore MarginallyUseful .

18


http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/madonna?http://www.youtube.com/user/madonna?http://www.youtube.com/user/madonna?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?https://www.youtube.com/watch?v=JF8BRvqGCNshttps://www.youtube.com/watch?v=JF8BRvqGCNshttp://www.youtube.com/user/rihanna?http://www.youtube.com/user/rihanna?http://www.youtube.com/user/rihanna?


19/64

QueryChannel-Seeking?

ResultTarget ofthe Query?

Rating Explanation

[skateboardingtwins official]

Yes

The PuehseTwins'SkateboardingChannel

Yes Very Useful

The Puehse twins are a pair of particularlywell-known skateboarding twins. Evenwithout their name in the query, researchshows that they are the target of the query.

This is their official skateboarding channel.

[puehse twinschannel]

Yes

The PuehseTwins'SkateboardingChannel

Yes Very Useful The Puehse twins are well known for theirskateboarding. This channel is the targetof the query.

[puehse twinschannel]

YesThe PuehseTwins’ TennisChannel

Likely orpossible,but notcertain

Useful

The Puehse twins have taken up tennis.This is their tennis channel. This is not thedominant interpretation of the query, but itis somewhat likely.

[CJTClol] Yes CJTClol’sChannel

Yes Very Useful

Even though the query doesn’t contain

“channel” or “official,” it is navigational.There are no reasonable interpretations forthe query other than this YouTubechannel.

[st mary’scollege official]

Yes

St. Mary’sCollege ofCaliforniaChannel

St Mary’sCollege ofMarylandChannel



Useful

Useful

The query is navigational, but we can’t tellwhich St. Mary’s channel the user islooking for. Channels for both colleges areUseful .

[Yosemite] NoYosemiteNational ParkTopic hannel


Useful

This topic channel is Useful because it isrelated to the query, the ratio ofgood-to-bad videos is high, and the goodvideos are easy to find.

[dogs] NoCoyote Topic Channel


MarginallyUseful

Even though the videos are high quality,they have just a marginal connection to thequery. The vast majority of users whoissue the query are looking for videos ofhousehold dogs, not coyotes that live inthe wild.

Please note: Even though topic channels are created differently, they are rated in the same way as other channel

results.

19


http://www.youtube.com/topic/lhJQqib8bvI/gray-wolfhttps://www.youtube.com/channel/UCz7vWYmm1ZIz4m-WphbPR0ghttps://www.youtube.com/channel/UCz7vWYmm1ZIz4m-WphbPR0ghttps://www.youtube.com/channel/UCz7vWYmm1ZIz4m-WphbPR0ghttps://www.youtube.com/channel/UCz7vWYmm1ZIz4m-WphbPR0ghttp://www.youtube.com/topic/LOWWLdWAnmY/yosemite-national-parkhttps://www.youtube.com/channel/UCrQ8BMjljGKzRH4bY2kXKYQhttps://www.youtube.com/channel/UCrQ8BMjljGKzRH4bY2kXKYQhttps://www.youtube.com/channel/UCrQ8BMjljGKzRH4bY2kXKYQhttps://www.youtube.com/channel/UCrQ8BMjljGKzRH4bY2kXKYQhttp://www.youtube.com/user/smcm?http://www.youtube.com/user/smcm?http://www.youtube.com/user/smcm?http://www.youtube.com/user/smcm?http://www.youtube.com/user/smcm?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/CJTClol?http://www.youtube.com/user/CJTClol?http://www.youtube.com/user/CJTClol?http://www.youtube.com/user/puehsetennis?http://www.youtube.com/user/puehsetennis?http://www.youtube.com/user/puehsetennis?http://www.youtube.com/user/puehsetennis?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?


20/64

4.3. Utility Rating for Playlist Results

When evaluating playlist results, you are not required to watch all of the videos in the playlist.

You will form an opinion of the videos in the playlist by watching parts of the first three videos

that play when you

click on the title or thumbnail in the rating task. Please base your rating on these three videos:

● The video that plays when you click the title or thumbnail in the rating task, and

● The following two videos that appear in the playlist right hand side feeder.

Please ignore the remaining videos in the playlist.

It is possible that one or more of the three videos you should watch doesn’t load or fails to play. If this happens,

please do the following:

● Flag the playlist as Did Not Load to indicate there is a problem with at least one of the videos.

● Watch the other videos and assign a rating according to the guidelines below.

Important: Please do not watch any extra videos to compensate for videos that do not load.

Evaluating Playlist Results

The first step in evaluating a playlist result is to determine whether the query is “playlist-seeking” or

“non-playlist-seeking”.

● Playlist-seeking queries. When the query includes words like “playlist”, “songs”, “top hits”, or “videos”, we

will assume the user is looking for a playlist result. Notice that “songs”, “top hits”, and “videos” are pluralized,

which indicates the user is looking for more than one song or video.

● Non-playlist-seeking queries. We will assume that the user is not looking for a playlist result when the

above words (or similar types of words) are not included in the query.

20



21/64

You will assign a utility rating to playlist results for playlist-seeking queries according to the following scale:

Rating Description

Very Useful Use this rating when all or most of the three videos you watch are Very Useful for the query.

Useful Use this rating when all or most of the three videos you watch are Useful for the query.

Marginally Useful Use this rating when all or most of the three videos you watch are only Marginally Useful for the query.

UselessUse this rating when all or most of the three videos you watch are Useless to the query and most userswould wonder why they are being shown the playlist result.

You will assign a utility rating to playlist results for non-playlist-seeking queries according to the following scale:

Rating Description

Very Useful Please do not use this rating when the query is non-playlist-seeking.

Useful Use this rating when all or most of the three videos you watch are Very Useful for the query.

Marginally Useful Use this rating when all or most of the three videos you watch are Useful for the query.

Useless Use this rating when all or most of the three videos are Useless for the query.

Please note that the rating standards are high when the query is non-playlist-seeking. To be rated Useful, all or most of the three individual videos you watch must be

Very Useful . To be Marginally Useful, all or most of the three videos you watch must be Useful.

21



22/64

4.3.1. Playlist Result Utility Rating Examples

QueryPlaylist

Seeking?

Playlist

Result

Description Rating Explanation

[Madonna

playlist]Yes

Madonna

YouTube

Playlist

Result 1

The main video is Madonna’s

“Like a Virgin” video. The

next two videos are her

“Express Yourself” and

“Erotica” videos.

Very

Useful

Since the query has “playlist” in it,

we know it is playlist-seeking. The

main video and the next two videos

are Madonna songs. Most users

would consider these videos Very

Useful or Useful.

[Madonna

songs]Yes

Madonna

YouTube

Playlist

Result 1






Very

Useful

Since the query has “songs” in it, we

know it is playlist-seeking. The main

video and the next two videos are

Madonna songs. Most users would

consider these videos Very Useful

or Useful.

[Madonna] No

Madonna

YouTube

Playlist

Result 1






Useful

For non-playlist-seeking queries,

Useful is the highest rating that can

be assigned.

[Madonna

songs]Yes

Madonna

YouTube

Playlist

Result 2

The main video is a video of

an interview Madonna did on

a TV show. The second

video is a promotional

documentary of “W.E.,” a

movie directed by Madonna.

The third video is a trailer for

the “W.E.” movie.

Useless

Even though the query is

playlist-seeking and the result is a

playlist, this particular playlist result

is Useless for the query. User intent

is to watch videos of Madonna’s

songs, not watch interviews,documentaries, etc.

[Madonna] No

Madonna

YouTube

Playlist

Result 2

The main video is a video of

an interview Madonna did on

a TV show. The second

video is a promotional

documentary of “W.E.,” a

movie directed by Madonna.

The third video is a trailer for

the “W.E.” movie.

Marginally

Useful

Most users would be looking for

music videos and not interviews,

documentaries, etc. All of these

videos are Marginally Useful for the

user intent, and because the query is

not playlist-seeking, the rating is

demoted one level.

[sesame

street

songs]

Yes

SesameStreet

Songs

Playlist

Result

The main video

and the nexttwo videos are music videos

from the children’s show

“Sesame Street.”

Very

Useful

Since the query has “songs” in it, we

know it is playlist-seeking. The mainvideo and the next two videos are

songs from the children’s show

“Sesame Street.” Most users would

consider these videos Very Useful

or Useful.

22


http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906E


23/64

[sesame

street ]

No

Sesame

Street

Songs

Playlist

Result

The main video

and the next

two videos are music videos

from the children’s show

“Sesame Street.”

Useful

For non-playlist-seeking queries,

Useful is the highest rating that can

be assigned.

4.4. Utility Rating for Movie Results

The first step in evaluating a movie result is to determine what the query is looking for. In some cases, the query is

looking for a particular movie. In other cases, the query is looking for a celebrity or a subject. You must try to imagine

user intent when assigning a rating to movie results.

You will assign a utility rating to movie results according to the following scale:

Rating Description

Very Useful Use this rating when you are certain that the user is looking for the movie.

UsefulUse this rating when the movie has a clear connection to the query, and is a result that some userswould be pleased to see but would not appeal to as wide an audience as a Very Useful movie.

Marginally Useful

Use this rating when the movie has a connection to the query, but it would not appeal to the vast majorityof users who issue the query.

This rating is also appropriate for movie results that serve a minor interpretation of the query.

Useless

Use this rating:

● When the movie has no clear connection to the query and would evoke the response “I have noidea why they are showing me this!” in the vast majority of users.

● The movie does not play. You will also check the Did Not Load play flag.● The video is unratable because it is in a foreign language and you are unable to determineutility. You will also check the Foreign Language flag.

This rating also applies when you cannot determine the utility of the movie within the allotted time.

IMPORTANT: Do NOT use this rating when the movie requires payment to play. For movies that do notplay without payment, please assume that the movie is what it claims to be.

Important: You are not required to assign the Useless rating to foreign language movies. If you believe that a foreign

language movie may have some utility, you may select another rating. Here are two possible scenarios:

● The user is looking for a foreign language movie. For the query [Die Legende von Paul und Paula],

English (US)

this German movie is exactly what the user is looking for. In fact, this movie is

Very Useful even though it is in a foreign language.

23


https://www.youtube.com/watch?v=7bBtTgzaF4khttp://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0


24/64

4.4.1. Movie Result Utility Rating Examples

Query URL Rating Description

[fletch]http://www.youtube.com/movie?v=-ssId2r7F1I

Very Useful

This movie matches the query. It is what the user is

looking for.

This movie requires payment. Although we can’twatch the movie without paying, we will assume thatthe movie is what it claims to be.

[gone in 60seconds]

http://www.youtube.com/movie?v=vt1AFkNitcU&ob=av1n&feature=mv_sr

Very Useful

This movie matches the query. It is what the user islooking for.

This movie requires payment. Although we can’twatch the movie without paying, we will assume thatthe movie is what it claims to be.

[police]

http://www.youtube.com/

movie?v=QuHMBFTBD5s

MarginallyUseful

This movie has a connection to the query, but it wouldnot appeal to the vast majority of users who issue the

query. Most users who issue this query are notlooking for a movie; they want to watch videos of theband “The Police”.

4.5. Utility Rating for Pay-Restricted Results

If the result requires payment, you are not required to pay and are not required to watch. We still ask that you rate the

block, even without the ability to view the video. Use the title, snippet, and thumbnail to give your best guess. Do not

assign the "Did Not Load" flag to pay-restricted results. (The Did Not Load flag signals to us that there may be

a problem with the result, but pay-restricted results typically function as we intend them to function.)

4.5.1. Pay-Restricted Result Utility Rating Examples

Query [adventure time]

Pay-RestrictedResult Block

Rating Useful

DescriptionThe user seeks episodes or clips from the Adventure Time TV show. The video uploader(CartoonNetworkEps) seems like the official owner of this TV show and this seems like an episode from theshow. Many users would find this result appealing if it were free, so it should be rated Useful.

Query [adventure time jiggler]

24


http://www.youtube.com/movie?v=QuHMBFTBD5shttp://www.youtube.com/movie?v=QuHMBFTBD5shttp://www.youtube.com/movie?v=QuHMBFTBD5shttp://www.youtube.com/movie?v=QuHMBFTBD5shttp://www.youtube.com/movie?v=vt1AFkNitcU&ob=av1n&feature=mv_srhttp://www.youtube.com/movie?v=vt1AFkNitcU&ob=av1n&feature=mv_srhttp://www.youtube.com/movie?v=vt1AFkNitcU&ob=av1n&feature=mv_srhttp://www.youtube.com/movie?v=vt1AFkNitcU&ob=av1n&feature=mv_srhttp://www.youtube.com/movie?v=-ssId2r7F1Ihttp://www.youtube.com/movie?v=-ssId2r7F1Ihttp://www.youtube.com/movie?v=-ssId2r7F1I


25/64


Rating Very Useful Description The user seeks a particular episode of this TV show. This is exactly what the user is looking for.

Query [adventure time jiggler]


Rating Marginally Useful

DescriptionThe user is looking for a particular episode of this TV show. This is the wrong episode, so it will have limitedappeal.

Query [adventure time jiggler free]

Pay-RestrictedMovie ResultBlock

Rating Useless to Marginally Useful

Description The user seeks a particular episode of this TV show that can be viewed without charge. Thispay-restricted result has limited appeal.

25



26/64

5. Assigning Quality Rating

The goal of YouTube Quality rating is to evaluate how well a YouTube page achieves its purpose. Unlike most web

pages, most YouTube pages only have one or two main purposes:

1. to provide a video that users will want to watch

2. to help users decide what video to watch next

To assign a YouTube quality rating you will need to know the purpose of the page. Here are the main purposes for

various result types.

Result

type

Purpose Example

Video 1,Section 5.1 for rating guidelines

Playlist 1 & 2,Section 5.2 for rating guidelines

Channel 2,Section 5.3 for rating guidelines

Pay-restrictedvideos andmovies

1,Section 5.4 for rating guidelines

5.1. Quality Rating for Video Results

If the video does not load or play for you, assign the Lowest possible quality rating.

Guiding principle when evaluating video quality: did content require effort and skill to produce?

You cannot be an expert judge of content quality for every niche interest on YouTube, but you can probably make a

guess about how much effort or skill was required to produce a video or collection of videos. Effort and skill is a useful

indicator of content quality so rely on this heavily when assigning Quality ratings.

26



27/64

Neutral or Worthless Indicators of quality (these simple counts can be aggregated by machines. If we wanted to

count these things we would ask a machine, not you!):

● View Count

● Subscriber Count

● Likes/Dislike count or ratio

Negative indicators of effort and skill include:

Copied content It is easy to upload someone else’s video (or a movie or TV show

you didn’t make). If the channel on which the video was uploaded

does not seem like an official or authoritative source for the

content, it is probably copied content. NOTE: parodies involving

subtle or clever audio/video editing DO take effort to produce and

do not deserve a harsh penalty if they seem well-done.

Low-quality, blurry, grainy or

scratchy audio or video

N/A

Distracting video annotations N/A

Useless video metadata For example, keyword stuffing or links to spammy websites.

Positive indicators of effort and skill include:

High “production value” Videos that show evidence of post-processing, video editing, or

sound mixing. For example, multiple camera shots in a single

scene or impressive special video or audio effects.

High quality, crisp and clearaudio and video

N/A

A strong narrative or plot; story

development; multiple scenes

N/A

Effort from multiple people to

produce.

For example, if a large cast of actors actively participates in the

video.

Use of stage props. Did the video uploader build, buy, or find something impressive,

expensive or rare? Compare this video of a rocket made out of

wooden matches and this video showing testing of a

machine-milled steel rocket nozzle in a rocket lab: the props in the

latter video are harder to make.

Useful video metadata. For example, lyrics to songs or links to related content. NOTE: not

all video metadata is useful. For example, keyword stuffing and

links to spammy sites are not helpful.

27


https://www.google.com/evaluation/url?q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DndgvIbtZ5hUhttps://www.google.com/evaluation/url?q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DfDkiuu1pALA


28/64

Quality indicators not related to effort and skill include:

Authority, expertise, or fame Videos from an official source (like the official White House

channel) or an authority (like the Mayo Clinic) or a famous source

(like a pop star) are often perceived as high-quality.

Misleading video thumbnailimages or video titles

These are indicators of low-quality videos. Users do not want to

be tricked or misled. If the video thumbnail and title seem to

advertise something the video does not provide, users will not be

satisfied.

Strong sentiment directed

towards content in user

comments

If there are a lot of comments about content that express strong

feelings, positive or negative, this may be evidence that the

content was engaging and generated an emotional response,

which may be a sign of high quality. However, comments directed

towards other commenters or to anything other than the content,

no matter how vehement, do not provide much evidence about

content quality.

Unique access Video of an interesting recent newsworthy event, for example

video of a disaster or protest, can have high quality if it is the only

video footage of its kind. For example, this video shows a plane

crash (warning: fatal accident footage). This footage is difficult to

obtain, so low audio and video quality can be somewhat excused.

Your gut feeling You may have watched a lot internet videos, as well as TV shows

and movies. Based on your personal experience, do you think

this video will appeal to users? If you have a strong opinion you

can allow it to influence your rating, especially if the video is

related to your interests.

5.1.1. Video Result Quality Rating Examples

Example 1

result block

28


https://www.youtube.com/watch?v=AtG3RUindC4


29/64

landing page

URL https://www.youtube.com/watch?v=Lm1fya3Qx4s

content No useful content: just static images with no relation to the title.

qualityfactors

● The video required little effort to produce.● Result title and thumbnail are misleading/deceptive.

29


https://www.youtube.com/watch?v=Lm1fya3Qx4s


30/64

● Audio/image quality are low because there is no audio andimages are static.

● Video description is not useful. The “click here” link provided inthe description is deceptive.

● The video content is not provided by an authority or expert.

quality ratingLowest

Example 2

resultblock

30



31/64

landingpage

URL https://www.youtube.com/watch?v=KIdTKE5g6bc

content Man remarks that a soccer game was well-played.

qualityfactors

● The video was not made with care or effort.● Result title and thumbnail are misleading/deceptive.● Image/audio quality is very low.● Video description is distracting and unhelpful.● The video content does not come from an authority or expert.

31


https://www.youtube.com/watch?v=KIdTKE5g6bc


32/64

qualityrating

Lowest

Example 3

result

block

landingpage

URL https://www.youtube.com/watch?v=3vE2WABErEM

32


https://www.youtube.com/watch?v=3vE2WABErEM


33/64

content A compilation of short amusing video clips.

qualityfactors

● The video is a compilation of videos from other users. Selectingthese videos and editing them into this single video probablyrequired some time and effort.

● Result title is okay but thumbnail is misleading and deceptive. Itsuggests that the video has racy content but there is very littleracy content in the video.

● Image/audio quality is fair. The “annotation” overlays/framesseem distracting and unnecessary.

● Video description contains some links and info of moderate valueand some keyword stuffing near the bottom.

● The video content is not produced by an authority or expert.

qualityrating

Low

Example 4

resultblock

33



34/64

landingpage

URL https://www.youtube.com/watch?v=1-sv8JGEdYw

content Clips of women and girls failing at various tasks.

qualityfactors

● The video is a compilation of videos from other users. Selectingthese videos and editing them into this single video probably

required some time and effort.● Result title and thumbnail are deceptive: they advertise racy

content but the video is not very racy.● Image and audio quality is poor.● Description consists mainly of keyword stuffing.● The video content is not produced by an authority or expert.

qualityrating

Low - Lowest

34


https://www.youtube.com/watch?v=1-sv8JGEdYw


35/64

Example 5

resultblock

landingpage

URL https://www.youtube.com/watch?v=VuAo_YpZk8A

content Clips of people failing at various tasks.

35


https://www.youtube.com/watch?v=VuAo_YpZk8A


36/64

qualityfactors

● The video is a compilation of videos from other users. Selectingthese videos and editing them into this single video probablyrequired some time and effort.

● Title and thumbnail are mostly accurate. The thumbnail imagescene is not present in the video but it is suggestive of the videocontent.

● Image and audio quality is mostly good.● Description consists mainly of keyword stuffing but the three links

it provides may be of some use.● The author has a website devoted to funny “fail” videos, with

content stretching back to 2009. We may regard him or her as aminor authority on fail videos.

qualityrating

Medium or Low+

Example 6

result

block

36



37/64

landingpage

URL https://www.youtube.com/watch?v=5Cod_wgM7n0

content A fan talks about an episode of a cartoon.

qualityfactors

● Audio is corrupted by white noise in several places in the video.The image quality is not very good.

● The author did not make this video with effort and care, otherwisehe would have fixed the audio issues.

● Result title suggests that the video is an actual episode of a cartoonrather than a review of the cartoon. The thumbnail is accurate.

● Video description is essentially empty.● The video content is not from an expert or authority.

quality

ratingLowest

Example 7

37


https://www.youtube.com/watch?v=5Cod_wgM7n0


38/64

resultblock

landingpage

URL The video was removed from YouTube due to copyright violations.

content A complete episode of a cartoon shrunk to fit inside a frame or border.

qualityfactors

● This is directly copied content - it required little effort to produce.● Result title and thumbnail are accurate.● Image quality is marred by the distracting frame or border.

● The description has keyword stuffing.● The user who uploaded the video is not an authority or expert.

qualityrating

Lowest

Example 8

38



39/64

resultblock

landingpage

URL https://www.youtube.com/watch?v=i6anvsoeOGw

content A short clip from a TV show.

qualityfactors

● This is copied content. However, the user who uploaded thevideo selected this particular scene for a particular purpose (i.e.to provide a preview) rather than uploading the entire TV show

episode, so it does not deserve the “lowest” quality rating justbecause it is unoriginal.● Result title and thumbnail are accurate.● Image and audio quality are poor.● The description is brief but accurate. It does not improve or

harm quality.● The video is provided by a devoted fan on a channel containing

content related only to this TV show. The user who uploaded the

39


https://www.youtube.com/watch?v=i6anvsoeOGw


40/64

video may be generously regarded as a minor authority orexpert.

qualityrating

Low - Medium

Example 9

resultblock

landingpage

URL https://www.youtube.com/watch?v=FRTf3UXCpiE

content The corporate anthem of Fujitsu Ltd. with lyrics slideshow.

qualityfactors

● This seems like copied content. However, finding this obscure oldsong and adding a lyrics slideshow requires some effort and care.

40


https://www.youtube.com/watch?v=FRTf3UXCpiEhttps://www.youtube.com/watch?v=FRTf3UXCpiE


41/64

● Result title and thumbnail are accurate.● Image quality is fine, but the audio is a little scratchy.● The description includes helpful information about the singer.● Little information about the user who uploaded the video is

available. There’s no reason to think that the user who uploadedthe video is an authority or expert.

qualityrating

Medium - High

Example 10

resultblock

41



42/64

landingpage

URL https://www.youtube.com/watch?v=dys8KUnwGGg

content A moaning “Ahhh” noise set to jarring music and disturbing images.

qualityfactors

● The author seems to have animated this video and made thesoundtrack on his own. This required effort and care.

● Result title and thumbnail are accurate.● Image and audio quality are good.

42


https://www.youtube.com/watch?v=dys8KUnwGGg


43/64

● The video description contains links and facts that may behelpful.

● The author is an authority or expert for this video because he hasbuilt a brand around this type of content, including a channel ofsimilar videos and an online store. His Twitter feed says “I writesongs that drive you insane.”

qualityrating

High - Highest

Example 11

resultblock

landingpage

URL https://www.youtube.com/watch?v=61G8iXGuSUg

content Software tutorial

43


https://www.youtube.com/watch?v=61G8iXGuSUg


44/64

Qualityfactors

● The video required care and effort to produce. The authordemonstrates his expertise about the software and usedartistic/design skills in the video as well.

● Video title and thumbnail image are accurate.● Image quality is fine. Audio could be a bit clearer or louder.● Description contains links that may be helpful.● Because the author has a YouTube channel and a separate

website devoted to this type of software tutorial he can beregarded as a minor authority or expert on this type of content.

qualityrating

High

Example 12

resultblock

44



45/64

landingpage

URL https://www.youtube.com/watch?v=LMs-VIbc_48

content “Trip to the esthetician” role playing video in ASMR style

Quality

factors

● The video seems to have required effort and care to produce. It was

done in a single 40 minute shot. The author paid careful attention tolighting and audio -- note especially the separate left and right audiotracks.

● Result title and thumbnail are accurate.● Image and audio quality are excellent.● Video description is extensive.

45


http://en.wikipedia.org/wiki/Autonomous_sensory_meridian_responsehttps://www.youtube.com/watch?v=LMs-VIbc_48


46/64

● The author is an authority or expert in this area because she makesmany of these videos, and according to her webpage, people donatemoney to support her to do this.

qualityrating

High - Highest

Example 13

resultblock

46



47/64

landingpage

URL https://www.youtube.com/watch?v=T_vGAPvjOc8

content Man narrates a video game adventure

quality

factors

● The video required some planning, effort and editing to create

and narrate.● Result title and thumbnail are accurate.● Image and video quality are excellent.● Description contains links that may be helpful.● The author is an authority or expert in this area because he has

built a brand (including an online store) and a following aroundthis type of video game adventure video.

47


https://www.youtube.com/watch?v=T_vGAPvjOc8


48/64

qualityrating

High - Highest

Example 14

result

block

landingpage

URL https://www.youtube.com/watch?v=-3a2qoyONVA

content Music video

qualityfactors

● The video required care and effort to create. It seems to haverequired filming at multiple locations and lots of editing and audioprocessing.

● Result title and thumbnail describe the video well.● Image and audio quality are good.● The description is sparse but accurate.

48


https://www.youtube.com/watch?v=-3a2qoyONVA


49/64

● RobynVEVO, the channel that hosts this video, is the primaryauthority on YouTube for music videos from the artist Robyn.

qualityrating

Highest

Example 15

resultblock

landingpage

URL https://www.youtube.com/watch?v=V2MBiS1kc_0

content Instructions on how to perform a neurological examination.

qualityfactors

● This seems to be a reliable instructional video produced by amedical professional. It requires a unique skill -- knowledge ofneurological examination techniques -- to produce.

49


https://www.youtube.com/watch?v=V2MBiS1kc_0


50/64

● The instructor in the video is a board-certified nurse practitionerwith a doctorate in nursing practice. She is an expert in thisarea.

● Result title and thumbnail are accurate.● Video audio and image quality are good.● Video description is sparse but accurate.

qualityrating

Highest

5.2. Quality Rating for Playlist Results

The purpose of a playlist is to provide the user with something to watch right away and a good collectionof videos to watch next. Ideally, the quality rating for a playlist would be the average of the quality ratingsfor the videos it contains plus or minus an adjustment for the cohesiveness, variety, order, size, andplayability of the collection considered as a whole. This ideal is impossible to achieve because you will

not have time to evaluate the quality of all the videos in a playlist. When evaluating a playlist, please dothe following two things, then make your best guess about what the ideal quality rating would be.

1. Skim the first three playlist videos to get an idea about Video Quality. An image below showswhere to find the first three videos of the playlist.

2. Visit the playlist list view and skim titles and thumbnails. The first screenshot below shows wherethe link to list view can be found, and the second image shows what list view looks like. Look atthe list view and ask yourself if the playlist seems to contain a good collection of videos, keepingin mind the following points:

a. cohesiveness is necessary: videos in a good playlist must have an obvious commontheme or topic -- like “amazing soccer goals” or “15th century lute music” or “birds ofprey”, and the title of the playlist should describe the topic.

b. variety helps the quality of a playlist, so long as the added variety does not causedepartures from the theme or topic.

c. order is important: if there is a “right” order for videos (like Episode 1, Episode 2, Episode3 of a TV show, or Acts 1, 2, 3 of a play) then playlists should put videos in the right order.

d. size helps quality: longer playlists are generally better than shorter playlists. However,length is less important than cohesiveness, variety, and order.

e. videos should be playable: sometimes playlists contain videos that no longer play, orare personalized and play only for a select group of people designated by the person whouploaded the video. These are obvious in playlist list view; see the second image belowfor example.

50



51/64

Playlist List view: displays all the videos in the playlist:

51



52/64

5.2.1. Playlist Result Quality Rating Examples

Example 1

resultblock

52


https://www.google.com/evaluation/portal/portal_files/VideoResultRating.pdf


53/64

playlist listpage

URL https://www.youtube.com/playlist?list=PLJvQXRgtxlulYgB1tznRR0h8biN2QCiM7

content Footage from Metallica concerts from 2014.

qualityfactors

● Audio quality is generally good, which is important formusic-related videos.

● There is high production value: multiple shots and evidence ofvideo post-processing.

● The playlist and videos come from the official Metallica YouTube

channel.● Unique access: videos show backstage footage of the band, which

would would be difficult for most people to obtain.● The playlist is organized around an obvious theme: Metallica

concerts in 2014.● Videos are arranged in an obvious order: date of concert.

53


https://www.youtube.com/playlist?list=PLJvQXRgtxlulYgB1tznRR0h8biN2QCiM7https://www.youtube.com/playlist?list=PLJvQXRgtxlulYgB1tznRR0h8biN2QCiM7


54/64

qualityrating

Highest

Example 2

result

block

landingpage

URL https://www.youtube.com/playlist?list=PLoip94oFJ4XuhyreC_ckBvihCya2CmLaB

content List of bad or funny auditions, typically a few minutes long, from variousTV-audition shows.

qualityfactors

● Many of the videos lack audio-visual crispness.● Many of these are uploaded by non-official uploaders (like

“shaunuk21”) and are thus copied content, but some areapparently from authoritative sources (“Britain’s Got Talent”, “The X Factor UK”).

● The collection as a whole is not organized around a tight theme:there are singing audition clips and music audition clips fromseveral years.

● The collection is not particularly comprehensive.

54


https://www.youtube.com/playlist?list=PLoip94oFJ4XuhyreC_ckBvihCya2CmLaBhttps://www.youtube.com/playlist?list=PLoip94oFJ4XuhyreC_ckBvihCya2CmLaB


55/64

qualityrating

Medium

Example 3

result

block

landingpage

URL https://www.youtube.com/playlist?list=PLC944E2DDE82E9904

content Eight music videos from various country music artists.

qualityfactors

● Some of the videos have low quality due to issues aroundaudio-visual crispness (like the first video, a cell-phone video of alive concert), or copied content.

● The bigger issue is that the playlist content has little to do with

the playlist title: the title indicates that the list has to do with aparticular song by a particular artist, but most of the videos inthe list have little or nothing to do with that song.

qualityrating

Low

Example 4

55


https://www.youtube.com/playlist?list=PLC944E2DDE82E9904


56/64

resultblock

landingpage

URL https://www.youtube.com/playlist?list=PLF9D86C3DCFEDF1EE

content Official Lady Gaga music videos alternating with “Loading Video…” spamvideos (see screenshot).

qualityfactors

There are several copies of a video titled “Loading Video …”. This shortvideo has no content and seems intended to raise video view countstatistics for the uploader who curated this playlist: from the typicaluser’s perspective it is spam, with no value.

qualityrating

Lowest

5.3. Quality Rating for Channel Results

Channels show content uploaded by one particular video uploader, which may be an individual likeJenna Marbles or an entity like Sony Pictures Entertainment Group. To assign a quality rating to thechannel, click on the uploaded videos link (illustrated below), pick any three uploaded videos on the

56


https://www.youtube.com/user/SonyPictureshttps://www.youtube.com/user/JennaMarbleshttps://www.youtube.com/playlist?list=PLF9D86C3DCFEDF1EE


57/64

channel page and skim them to get an idea about the quality of the videos uploaded to the channel.Then make a guess about the average quality of the ratings on the channel, and assign that for a qualityrating.

5.3.1. Channel Result Quality Rating Examples

Example 1

resultblock

57




58/64

landingpage

URL https://www.youtube.com/user/NationalGeographic/videos

content Videos from National Geographic

quality

factors

This is the official channel for this well-known source of educational,

scientific and documentary content. Videos have high production quality,involve many people and props, and clearly took a great deal of effort toproduce. Audio visual quality is quite good.

qualityrating

Highest

Example 2

resultblock

58


https://www.youtube.com/user/NationalGeographic/videos


59/64

landingpage

URL https://www.youtube.com/user/FranzLisztFerentz/videos

content Classical music recordings

quality

factors

● The recordings have good audio quality.

● The videos seem to be copied content.

qualityrating

Medium

Example 3

resultblock

59


https://www.youtube.com/user/FranzLisztFerentz/videos


60/64

landingpage

URL URL not given: this content is in violation of YouTube terms of service and

will be removed.

content Full-length movies

qualityfactors

● These full-length movies have low audio-visual quality, manydistracting video annotations, keyword stuffing in video metadata,and they are copied content.

● The channel has uploaded only these three videos.● The channel name suggests that all videos will be horror movies, but

this is not the case.

quality

rating

Lowest

5.4. Quality Rating for Pay-Restricted Movies and Videos

Always assign the highest quality rating to pay-restricted movies and videos.

5.4.1. Channel Result Quality Rating Examples

Example

resultblock

60




61/64

landingpage

URL https://www.youtube.com/watch?v=sIbhaVkWDes

content Pay-restricted cartoon episode.

qualityfactors

Please assume that pay-restricted content is highest-quality.

qualityrating

Highest

61


https://www.youtube.com/watch?v=sIbhaVkWDes


62/64

6. Assigning Flags to YouTube Result Blocks

You will assign the following flags as needed: Porn / Offensive, Foreign Language, Did Not Load.

6.1. Assigning flags to video results

Porn / Offensive: Use this flag when the content of the video result is offensive or shocking because of sex-related

content, violence, bad language, etc.

Foreign Language: Use this flag for videos that would be regarded as foreign by most users in your locale. In

general, the Useless rating is appropriate for foreign videos unless you think the video has some utility despite being

foreign. In that case, please assign a rating from the rating scale.

Did Not Load: Use this flag when the video does not play for any reason, e.g. taken down, marked as private,

prohibited for the locale, etc. Do NOT use this flag for paid results, which typically function as intended after payment.

6.2. Assigning flags to non-video results

If a result page contains multiple videos (such as playlists, channels), assign the appropriate flag when some of the

first few videos on the page are porn / offensive, foreign language or did not load.

7. Reporting Results with Duplicate Landing Pages

In addition to assigning utility ratings to each display block result, we would like you to tell us whether two results are

duplicates (dupes).

7.1. Pre-Identified Duplicates

Some duplicate results are automatically detected and will be pre-identified for you. They will be highlighted by a grey

or green background, and annotated by either "Same as..." text right below the flags and above the result block.

This is what a pre-identified “Same as…” result block look like:

Same as… pre-identified duplicate result

62



63/64

Please note:

● Results annotated by “Same as…” may

not be given different ratings/flags/comments. Their utility ratings, page quality ratings, flags, and comments will be automatically transferred to each other.

● You cannot un-check dupes that have been automatically detected and pre-identified.

7.2. Rater-Identified Duplicates

We would like your help identifying duplicate results that have not been automatically detected.

Encountering a list of identical or very similar results on the YouTube search results page is generally a bad

experience for users. Use your judgment when applying this definition: two video results are duplicates if they have

similar or identical content and the user who issued the query would not find the second video at all engaging after

having enjoyed the first to its fullest extent. A video result should NOT be marked as a duplicate of a playlist or

channel result.

7.3. Reporting Duplicate Results

When you notice that the results in two or more display blocks are duplicates, please click on the Report Dupe button

of one of the results. The current result (the result you clicked on) will be highlighted by a thick solid red border and the

button's name will change to Select Dupes. You can then check results that are duplicates of the current result, and

the checked results will be highlighted by a red dotted border.

The duplicate results that you checked will be annotated by "Dupe of..." text right below the flags and above the

display block. After selecting all dupes, please click the red Select Dupes button to return to the normal rating mode.

The button's name will change back to Report Dupe, and you will be able to report other sets of dupes (if there are

any). If you change your mind, you can always un-check a result. Please note that rater-identified duplicate results

annotated by "

Dupe of..." may be given different utility ratings/flags/comments.

Please identify dupes both within the same side and across sides. Even for cross-side results, you should still

ask yourself the question "Would you want to see both results if they were returned by the same search engine?"

Here are some examples:

Dupes

QUERY: [Princess Mononoke Trailer]

URL 1: http://www.youtube.com/watch?v=oZ4pxURy5-I

63


http://www.youtube.com/watch?v=oZ4pxURy5-I


64/64

URL 2: http://www.youtube.com/watch?v=pkWWWKKA8jY

Reason: These two video results are duplicates because they are movie trailers that differ only in aspect ratio. Even

though they were uploaded by different YouTube users and their run times are slightly different, they are duplicates.

Users would not want to see both results returned by YouTube.

Not Dupes

QUERY: [Por una cabeza]

URL 1: http://www.youtube.com/watch?v=ZbyIUd6tfp0URL 2: h ttp://www.youtube.com/watch?v=F2zTd_YwTvo

Reason: These videos show actors dancing the tango to “Por una Cabeza” from two different movies (“True Lies” and

“Scent of a Woman”). Even though the song is the same, the videos are very different. Users could benefit by

YouTube returning both videos for the query.

8. Assigning an Overall Side by Side Score (if requested)

When you are done assigning ratings and flags and flagging pages with duplicate landing pages, please decide which

side is better according to the SxS Guidelines .

Reminder : You will only be asked to assign an Overall Side by Side score in some Block Utility tasks.
https://www.google.com/evaluation/portal/portal_files/side-by-side.pdfhttp://www.youtube.com/watch?v=F2zTd_YwTvohttp://www.youtube.com/watch?v=F2zTd_YwTvohttp://www.youtube.com/watch?v=ZbyIUd6tfp0http://www.youtube.com/watch?v=pkWWWKKA8jY

Documents

YouTubeGuidelines.pdf