How Helpful is Helpful?A Design Study in Visualizing Reviews

Charles Hermann, Huisi Luo, Pratik Shekhar

College of Computing and Informatics Drexel University

Abstract: The goal of this study was to analyze Glassdoor reviews of six companies and look for trends in the reviews and ratings of these companies. Through a visual analysis with Tableau, Displayr, and Fusion Tables of the ratings on Glassdoor of each of these companies a few trends emerge. These trends pose more questions than answers. More importantly, the rating of “helpful” to reviews appears to be more problematic than helpful in analyzing reviews of these companies, which creates even more questions than answers through visual analysis.

Index Terms: Glassdoor, amazon, apple, facebook, google, microsoft, netflix, reviews, ratings, bias, implicit bias, explicit bias, online

—————————— ♦ ——————————

1 INTRODUCTIONOnline reviews have become increasingly influential for consumers.[1] These reviews influence the purchases consumers make, the services they acquire, and even the employers they choose. A recent survey found that 48% of job seekers have used Glassdoor at some point.[2] “The study also found that 60% of job seekers would not apply to a company with a one-star rating (on a five-point scale).”[2] It has been shown that online reviews tend to have a user bias from reviewers. One of these slants is highly intuitive: users tend to post reviews that are either highly positive or highly negative.[3]

Glassdoor’s mission is to “help people everywhere find a job and company they love.” (Glassdoor) The website was started in 2007 when Richard Barton and Robert Hohman were brainstorming for a new idea.[4] The two began discussing what may have happened when a raw data employee survey for their former employer, Expedia, had been made public.[4] They soon came up with the idea for a website in which employees could rate their employers and give feedback for potential new employees or job seekers. Since that day, Glassdoor has become a networking site where employees can rate and rank employers. They consider this a “‘give-to-get’ model.” Users have to “first contribute a review and/or salary survey for their current employer or any position they have had in the last three years,” and then they are able to access other reviews and salaries on the Glassdoor website.[3]

The dataset used for this analysis was retrieved from the website Kaggle. It was posted by Peter Sunga.[5] The dataset contains over 67 thousand reviews for six

companies on Glassdoor’s website.

1. Glassdoor Data on Six CompaniesThe dataset used to analyze the ratings of Glassdoor reviews contains reviews of multiple aspects of employment for these companies. In these ratings for companies of employees, a dataset of all ratings has been compiled. These ratings include the company names, locations, date of posting, job titles, summaries, pros and cons, “advice to management,” as well as overall ratings, work balance stars, culture value stars, career opportunities stars, comp benefits stars, senior management stars, and a count of how many found these reviews helpful.

The focus of this analysis was on the Company Name, the Work Balance, Culture Values, Career Opportunity, Comp Benefits, and Senior Management stars, as well as Overall Ratings and Helpful Count of each review. The star ratings for each of the respective categories is rated from one to five stars. The Helpful Count is a binary count of how many viewers found the reviews “helpful” (by clicking that they found the review helpful).

2. Helpful CountIn these ratings, it is observed that the rate at which users find reviews helpful has little in common with any other ratings. In the enumerative reviews of these companies, what stands out is the disconnect between the “helpful ratings” of the reviews. What needs to be investigated is how the helpful ratings align with other variables in the reviews. The Helpful Count was farther visualized to seek any correlation between reviews found helpful and the

particular types of ratings given for each company in their respective categories.

2 TOOLS Tableau is powerful yet has rich features, it is a visualization tool used for analyzing data and obtaining insight as well as patterns. The process of data analysis is very fast with Tableau and the visualizations created are in the form of dashboard and worksheets. It allows the user to create many worksheets to visualize as well as comparing different categories of the dataset by blending data, analyzing data in real time, and collaborating data. Displayr is data science software for Analysis and Reporting. We used this tool for Alluvial flow to show how different ratings and review interconnect. Fusion tables is a web service provided by Google for data management and visualization.

3 METHODS

3.1 Data Visualization ProcessIn the data visualization of the employment conditions, multiple types of visualizations were completed to develop a better picture of these reviews. The initial questions that were posed to analyze the data were to determine which company the best cultural values have, which company has the greatest employee benefits, and if these variables had a strong connection with overall reviews. Box-and-whisker plots were developed to display the six leading tech companies and focused on the overall ratings of each company. Stacked bars also made use of color coordinated bars denoting each company. Alluvial flows were utilized to develop an image of how these ratings may be interrelated. In noticing the disparities in the Helpful Count with the other ratings, the data was entered into Google Fusion Table to examine trends in the ratings.

3.1.1 Company ValueBox-and-whisker plots were utilized in analysis of which company has the best cultural values. (See Figure 1) The ratings were displayed in terms of high and low ratings with the points on the chart showing how many ratings were higher or lower in cultural values of the company.

The first company in the graph is Amazon, while viewing amazon one may notice that although there are many lower ratings, the density of ratings ran out around 2.8K. Also, the highest rating reached up to around 10K, which was the highest measurement on the charts. Amazons ratings are mainly contained in the upper Quartile which stretches from 3-4 rating, making it the highest rated company. Amazon also has many ratings scattered in between these highs and lows. The only other company to reach 10K in ratings was Apple. On the other side of the spectrum in comparison, Netflix hardly had its density reach 1K and had absolutely nothing above that mark. Making it possibly the worst company in terms of cultural value in overall ratings. Facebook was very similar to that of Netflix but

contained one small translucent dot portion worth of data just below 3K, making it just better than last. Apple and Google are very similar in terms of the overall span. Where they differ is apple contains the bulk of its rating in 3, where google contains its bulk above. In conclusion Amazon definitely has the best overall cultural value according to employee ratings, and although Apple reaches the highest point with some rating it as highest, consistency is lacking, and runner up is Microsoft. A few things to take into consideration with this type of visualization is that years are not taken into account and if they would be, there may be an entirely different output of data. Also, Netflix is relatively new to international standards compared to the other companies on the list, and this must be taken into consideration.

Figure 1 - Box-and-Whisker Plot, Company Value

The Box and Whisker plot displays the one through five stars in amounts through a gestalt visual interpretation that gives an overall picture without needing detailed descriptions. The online application of these charts gives more interactive information to the user with allows for greater detail. The captured image is limited in greater detail but still imparts the general information of distribution about the ratings for Company Value.

3.1.2 Company BenefitsIn the analysis of Company Benefits ratings in the Stacked Bar chart, the color coordinated bars denote each company. (See Figure 2) Amazon is the blue bar color. It can be easily viewed that most of the data for Amazon lies in the three to five-star ratings with the greatest magnitude found at four. The orange, Apple bar is similar to the Amazon data with ratings also residing in the three to five range but having the largest rating at five. Facebook, the red bar, are primarily located in the range of one and two with a slight number of occurrences at four and a large quantity at five. The turquoise, Google bar has a negligible amount of ratings at one and two, with more company benefits ratings located in three. The Google ratings increase substantially at four and five, with a majority located in the five-star rating. The green, Microsoft ratings appear to mirror the Amazon ratings but with less quantity of overall votes. Finally, the yellow, Netflix bar has the least number of total

votes. Some of these votes can be seen at three and four with most of them residing in the five-star rating.

Figure 2 – Stacked Bars, Company Benefits

Analyzing each color in this graph and focusing on comparative growth of colors, neglecting the individual size, Facebook appears to be the leader in company benefits. The Facebook company benefits ratings are mostly focused on the five-star ratings. The company most lacking in company benefits appears to be Amazon. The Amazon company ratings has the most visible one- and two-star reviews with a substantially larger three-star review as well.

3.1.3 Overall AnalysisIn this analysis, an alluvial flow was developed in Displayr to analyze all ratings. (See Figure 3) This visualization benefitted greatly from having an interactive view. In a static view, the alluvial flow is quite complicated. The flow of overall ratings from each company into corresponding star ratings at first glance appears more like a server mainframe than any type of flow of data. With the interactive abilities of Displayr in the development of the alluvial flow, one is able to highly each variable and display the precise data of each company and corresponding flow of data.

Figure 3 – All ratings, Alluvial Flow

The static alluvial flow still does display some interesting trends in the flow of ratings for each of these companies through the Glassdoor ratings system. The most notable display in this alluvial flow is the number of ratings for each company, with Amazon having a significantly larger portion of reviews (at 40% of all reviews). It is also quite clear that Netflix has a substantially lower amount of reviews in relation to the other five companies, with Facebook not far behind, with one percent and two percent respectively.

Another instance highly visible in this alluvial flow is the rate at which reviews are dispersed. Although Amazon has a massively larger amount of reviews, which Microsoft and Apple coming in second, the amount of low ratings for each company is significantly smaller than the high ratings. Facebook, with only 2% of all reviews, has a predominant high rating. Majority of all reviews for all six companies are situated within the four- and five-star ratings for majority of factors. Creating an alluvial flow containing only overall-ratings and helpful-count was much clearer than an alluvial flow with all rating data. In this alluvial flow, the interactive feature of Displayr was much less important. The overall ratings are clearly higher for all companies. A full 35% of five-star overall reviews dominates this representation, with 33% four-star ratings coming a close second. In this display, it is interesting to note that all six companies have nearly all positive overall reviews on Glassdoor.

Figure 4 - Overall Ratings/Helpful Count, Alluvial Flow

3.1.4 Helpful Count (the outlier)The display of Overall Ratings combined with the Helpful-Count in alluvial flow was quite interesting and warranted farther investigation. To visualize this relationship, the data was all displayed in Fusion Tables. This was displayed in a line graph with all ratings displayed as well as the representation of the Helpful-Count overlaid on the ratings. What emerged was a picture that showed Helpful-Count ratings appearing to be quite spurious. All of the ratings follow the same general trend, with the exception of the Helpful Count. As shown in the Fusion Table graph, the Helpful-Count appears to be quite arbitrary compared to all review ratings. (See Figure 5)

Figure 5 - Fusion Table generated line chart, Helpful Count and all values

3.2 ResultsWhen looking at these visualizations collectively, patterns emerge or fail to emerge. As discussed in the methods of visualizing this data, some companies benefit from a tremendously larger number of ratings. All companies appear to have more favorable ratings. This could mean that all companies have favorable work environments. It could also imply that there is some other moderating variable influencing these results.

From the Box-and-Whisker Plots and the Stacked Bar chart, it seems that Company Cultural Values and Company benefits do not seem to intertwine much and the company with the worst rating on value has the best benefits.  Amazon is leading the competition in Company value and Facebook is the leader in the company benefits category. Although Facebook is the leader in the Benefits category along with life and work balance, they remarkably lack in the company values section, which possibly keeps them away from the position of the best overall company. Closely rated is that of Microsoft because although it was never the best in any one particular category, it still was rated in a higher category in all categories than any other company, simply that it was always on the positive side of the spectrum.

When looking at the Alluvial Flow charts, it becomes clear that all companies have overall favorable ratings in all categories, as previously discussed. The amount and rate of reviews seems to only slightly alter the results of these ratings, pulling some ahead more than those with a smaller number of reviews.

Most importantly, the greatest insight comes when looking at the Helpful Count of reviews and ratings, there seems to be no correlation at all to any of the other ratings.

4 DISCUSSIONThe most emergent trend, as repeatedly elucidated throughout this analysis has been the disparity in numbers of reviews between these six companies and the tendency of ratings to be positive. Several aspects of online reviews and the use of Glassdoor itself may have an impact on these trends. Firstly, there has been a considerate amount of attention paid to the lack of anonymity in the reviews on Glassdoor. Courts have ruled that Glassdoor reviews are not protected from employers.[6] In April of 2017, courts ruled in a closed proceeding that Glassdoor reviewers “are not entitled to anonymity and that the government can compel the company to provide users’ information.”[6] Companies have the ability to demand to know the identity of those leaving reviews. This can have a considerably detrimental effect on the reporting of negative reviews. Not only can this cost employees their occupational status and/or income but can have legal ramifications that reach much farther. If these reviews are not anonymous, retaliation from companies that seek to protect their brand or status is likely.

Another variable that may interplay with the self-reporting of reviews is the incentivizing of reviews by companies. Business and journal articles have been written exploring the effective of providing incentives for employee reviews on sites such as Glassdoor.[7] Combined with the lack of anonymity, this would likely push a trend towards more high rated reviews.

In the discussion of the arbitrary nature of the “Helpful Count” in the ratings system on Glassdoor, it seems highly likely that this is a structural bias in the website itself. Users must contribute reviews to be able to view more than three reviews. This means that many displays of the helpfulness of a review can be a perpetuating bias, where the “helpful” rating of a document places it higher in the retrieval list and generates more “helpful” ratings. The display of reviews that are considered most helpful may create a perpetual motion of the generation of helpful counts.

Another issue that was observed is the visualizations themselves have a greater range of information via computer interaction where users are able to access more detail through interaction with the visual representations. This interaction is limited in picture format, but the visualizations included do demonstrate the information in an understandable scheme. In a textual format, the information gained from the visualizations is limited only to what can be immediately displayed. This limits the detail of the relationships between ratings. Introducing too many variables into a visualization would create a situation where the user would likely not be able to ascertain any information at all from the visualization. This lends to the idea that it is possible there is a higher degree of nuance in these ratings that is able to be represented here. Multiple representations were created that were discarded in this study because the ability to visually display differences were just not able to be presented in a static format and gave no particular insights.

5 CONCLUSIONIn visualizing the reviews for companies on Glassdoor, problematic themes arise from these insights that question the process of employee/employer reviews on this website. Questions remain unfulfilled as to the amount of bias in these reviews. From these visualizations, a large picture emerges that shows a need for a deeper investigation of this process of reviews. As these reviews can be detrimental to the companies being reviewed and the employees seeking better employment, it becomes necessary to have a better understanding of the process.

More investigation is needed to determine the accuracy of these reviews and analyze the impact of multiple moderating variables in the legitimacy of these reviews. It is also important to acknowledge that it is likely these companies simply have happy employees that tend to leave favorable reviews because they wish to convey that they are/were pleased with their current/former employer.

ACKNOWLEDGMENTSThank you to Peter Sunga for creating this dataset.

A special thanks to Dr. Chaomei Chen for giving the knowledge to be able to use the tools mentioned to complete this project.

REFERENCES[1] D. Hinckley, “New Study: Data Reveals 67% of Consumers are Influenced by Online Reviews,” Moz, 03-Sep-2015. [Online]. Available: https://moz.com/blog/new-data-reveals-67-of-consumers-are-influenced-by-online-reviews. [Accessed: 17-Mar-2019].[2] DeMers, “How Negative Online Company Reviews Can Impact Your Business And Recruiting,” Forbes, 28-Mar-2016. [Online]. Available: https://www.forbes.com/sites/jaysondemers/2014/09/09/how-negative-online-company-reviews-can-impact-your-business-and-recruiting/#2815f8f21d9b. [Accessed: 17-Mar-2019].[3] N. Klein, I. Marinescu, A. Chamberlain, and M. Smart, “Online Reviews Are Biased. Here's How to Fix Them,” Harvard Business Review, 14-Mar-2018. [Online]. Available: https://hbr.org/2018/03/online-reviews-are-biased-heres-how-to-fix-them. [Accessed: 17-Mar-2019].[4] N. Wingfield, “The Art of 'Something From Nothing',” The New York Times, 20-Dec-2017. [Online]. Available: https://www.nytimes.com/2014/04/14/technology/the-art-of-something-from-nothing.html. [Accessed: 17-Mar-2019].[5] P. Sunga, “Google, Amazon and more Employee Reviews,” Kaggle, 13-Dec-2018. [Online]. Available: https://www.kaggle.com/petersunga/google-amazon-facebook-employee-reviews. [Accessed: 17-Mar-2019].[6] K. Gorgone, “Don't Review Your Employer on Glassdoor Until You Read This!,” Schaefer Marketing Solutions: We Help Businesses {grow}, 25-Jul-2017. [Online]. Available: https://businessesgrow.com/2017/07/26/glassdoor/. [Accessed: 17-Mar-2019].[7] I. Marinescu, N. Klein, A. Chamberlain, and M. Smart, “Incentives Can Reduce Bias in Online Reviews,” Mar. 2018.