Massachusetts Institute of TechnologyDepartment of Electrical Engineering and Computer Science

Proposal for Thesis Research in Partial Fulfillmentof the Requirements for the Degree of

Doctor of Philosophy

Title: Crowd-Powered Interfaces

Submitted by: Michael S. Bernstein1 Ashburton Pl. #2Cambridge, MA 02139

Signature of Author: ________________________________

Date of Submission: May 10, 2023

Expected Date of Completion: June 2012

Laboratory where thesis will be done: CSAIL

Brief Statement of the Problem:This thesis investigates crowd-powered interfaces: interfaces that motivate and then embed human activity to support interactive applications. Crowd-powered interfaces recruit humans or use online datasets to create interactions that would be difficult using traditional techniques. For example, a crowd-powered interface using paid crowd workers can mark cuts and edits for a paragraph, then give users a slider to shorten their writing. We are building prototype systems to map out the design space of outsourced, friendsourced, and data mined interfaces. To do so, we are addressing technical and motivational challenges in hybrid human-algorithm systems.

IntroductionAlgorithms and design drive many innovations in interface technology, but each user is still an island. Word processors can help with layout, for example, but they can't tap deeply into our collective writing knowledge. This collective knowledge holds huge potential: other authors can provide writing and editing support for a paper, databases of professional writing can be data mined for patterns that novices might adopt, and a writer’s friends would know whether colleagues’ names are spelled correctly. This knowledge could grant interfaces powerful new capabilities.

This thesis contributes interfaces using crowd contributions to support complex authoring and information management applications. They are crowd-powered interfaces: interfaces that embed human-authored datasets or human computation [33] in a larger application. Crowd-powered interfaces introduce new interactions by motivating or paying human workers to complete tasks algorithms cannot do reliably. For example, paid crowd workers might suggest edits to tighten up an article’s wording, giving the user a slider to change an article’s length.

This thesis will introduce six research systems exploring crowd-powered interfaces:

1. Soylent, a word processor with a crowd inside. Soylent pays internet workers small amounts of money to support new interface features like distributed human proofreading, automatic text shortening and natural language macros.

2. The Surgeon’s Assistant, a Photoshop plug-in that pays crowd workers to help the user in real-time via a screen-sharing system.

3. Collabio, a social game that encourages members of an online social network site to tag each other with descriptive terms.

4. FeedMe, a news reader plug-in that encourages users to share web content with their friends and colleagues, then learns from the shared content to recommend new content.

5. PingPong++, a ping pong table that remembers every game played on it and uses past play statistics to recommend plays in realtime.

6. An Unnamed Data Remixing System, which allows novices to wire together expert-created content to create their own work.

In support of crowd-powered interfaces, we propose new algorithms and social incentives for crowd control. Lazy crowd workers will do as little work as possible, while overeager ones will make unexpected (or worse, undesired) changes. We introduce design patterns for controlling these problems. For example, we decompose the work into separate stages: finding problems, fixing problems, and quality control. Motivation is also a difficult challenge, so we introduce the use of social connectedness as an incentive.

Thesis StatementCrowd-powered interfaces can gather and mobilize online contributions to enable powerful new kinds of interactive authoring and information management systems in domains like word processing, image editing, web feed management, expert finding, and recommender systems.

Design SpaceCrowd-powered interfaces combine two user interfaces: the interface shown to the crowd, and the interface shown to the end user. The design space of crowd-powered interfaces must consider both user groups.

CrowdMotivation What incentive does the system use to encourage participation? Is the crowd

intrinsically or extrinsically motivated to participate?Quality Control How does the system automatically detect and filter out poor crowd

contributions?Crowd Size How many people are in the crowd?Temporality Is the crowd available live, or is the system mining previous contributions?Collaboration Can the crowd collaborate on the work, or is the work distributed and individual?Expertise Does the task require crowd members to have a specific expertise, like knowledge

of a particular subject? Or, can most Internet users complete the tasks?Task What task does the application need the crowd to help with?

UserDelay Is feedback immediate, or must the user wait for work to complete before the

crowd-powered interaction is available?Domain What task is the user trying to complete? For example, word processing, image

editing, and web search can be crowd-powered.Initiative Does the user explicitly request help, or is the crowd actively monitoring for

opportunities to step in?Feedback Can users tell the crowd whether the work was acceptable? Can the crowd

communicate with the users?

ApproachTo approach this design space, I will orient my thesis around three crowd-powered techniques. These approaches are outsourcing, friendsourcing, and data mining.

OutsourcingPaid crowd work has developed as a platform, and we can now build systems that permanently embed outsourced work in their interfaces. Services like Amazon Mechanical Turk1 can serve as prototyping

1 www.mturk.com

populations and are appropriate solutions when the designer doesn’t have any intrinsic motivators in mind. For example, Internet users might not be interested in proofreading a paper for free, but might be willing to proofread a small section for a little money. This crowd is potentially quite large, but spamming means that quality control becomes necessary. Paid crowd workers are almost always available, which makes them ideal for interfaces that need relatively low latency, on the order of minutes.

This thesis introduces Soylent, a word processor that embeds paid crowd workers to support writing tasks like text shortening, proofreading, and human-language macro tasks. We also plan to pursue an outsourced photo editing plug-in that places crowd workers in a realtime support position, so they are acting on a live version of the document rather than a batched-off request.

FriendsourcingCrowd-powered interfaces can find a crowd with relevant expertise if they use the correct motivational levers. Friendsourcing recruits the user’s friends, family and colleagues on a social network to share information about the user. This information can enable personalized interfaces like recommender systems, news filtering and expertise finders. In friendsourced interfaces, the crowd is relatively small, intrinsically motivated by social connections, and appropriate for tasks that need information about individuals in the social network.

The major challenge in friendsourcing is motivating participation. We have designed two systems, FeedMe and Collabio, to encourage members of a social network to share information about each other. These systems have gathered enough information to power personalized interfaces like RSS recommenders, expertise finders, and exploratory visualizations. The personalization data comes as a side effect of people in the social network communicating, sharing information and playing games.

Data MiningIn many cases, the crowd has already produced large datasets and activity traces that an interface can mine. For example, we can turn to large online databases of writing, photographs, videos and music to find raw material for creative authoring applications. Crowd-powered interfaces can remix this data to make it useful to an end-user who is creating new content.

We are not the first to consider putting crowd data into an interface. However, most of these interfaces consist of a single call-and-response: MySong [26] takes a vocal line as input and produces a backing track, HelpMeOut [13] takes in a compiler or runtime error and suggests a fix, and Google Suggest proposes a complete query given an in-progress query. My goal is to demonstrate that these interfaces can work in a tighter loop with the user, with continuous feedback. The specific application area in mind is again an authoring environment where amateur users can tie together small elements from expert-created material to produce an original piece – a digital mosaic, or found footage.

Related WorkCrowd-powered interfaces draw on related work in two areas: interfaces integrating crowd contributions, and research in online community contributions.

Crowd-powered InterfacesCrowd-powered interfaces are the logical extension of existing research that marries Big Data with interface design. Several such systems power novel interactions with the wisdom of crowds. These systems have typically focused on a single call-and-response interaction with the crowd data: the user’s current state produces one suggestion from the crowd server. HelpMeOut [13] collects debugging traces and applies others’ error solutions to help fix code. MySong [26] indexes a library of music chords to enable the user to build a chord progression by singing a melody line. Sketch2Photo [7] transforms a hand-sketched photo outline annotated with descriptive terms and produces a composite photo by searching a large database of images. Finally, Google Suggest mines the search engine’s query logs to speed and direct new queries.

Another set of systems have developed novel exploration interfaces for human-generated data. Mr. Taggy [16] indexed a social bookmarking site and to provide a relevance feedback interface for navigating the bookmarked sites. Sense.us [15] and ManyEyes [30] allow internet crowds to comment on and refine information visualizations. BLEWS [9] produced an exploratory visualization of internet blogs with respect to their linking patterns and the red/blue political spectrum.

We also build on work embedding on-demand human workforces inside applications and services. ChaCha2 recruits humans to do search engine queries for users who are mobile; Amazon Remembers uses Mechanical Turk to find products that match a photo taken by the user on a phone; Sala et al.’s PEST [25] uses Mechanical Turk to vet advertisement recommendations. These systems consist of a single user operation and little or no interaction. We extend this work to more creative, complex tasks where the user can make personalized requests and interact with the returned data by direct manipulation.

Friendsourced applications draw on years of CSCW research supporting workgroup practice. These systems facilitate workgroup awareness [11] and help employees find coworkers who can answer their questions [1]. However, these systems typically rely on each user investing effort into updating their own status or expertise in the system, or in installing logging software to do so semi-automatically. Friendsourced applications typically reverse the dynamic and recruit a user’s motivated friends rather than require the user to do work on their own behalf.

Crowds and ContributionsGathering data to train algorithms is a common use of crowdsourcing. Mechanical Turk is already used to collect labeled data for machine vision [28] and natural language processing [27].

Our reliance on human computation means that crowd-powered interfaces’ behavior depends in large part on qualities of crowdsourcing systems and Mechanical Turk in particular. Recently, Ross et al. found

2 http://www.chacha.com

that Mechanical Turk had two major populations: well-educated, moderate-income Americans, and young, well-educated but less wealthy workers from India [24]. Kittur and Chi considered how to run user studies on Mechanical Turk, proposing the use of quantitative verifiable questions as a verification mechanism [18]. Heer and Bostock explored Mechanical Turk as a testbed for graphical perception experiments, finding reliable results when they implemented basic measures like qualification tests [14]. Little et al. advocate the use of human computation algorithms on Mechanical Turk [21]. Quinn and Bederson have authored a survey of human computation systems that expands on this brief review [22].

Friendsourcing is inspired by prior work on human computation [31], which aims to obtain useful information for computers by enticing users to provide it. Games with a Purpose [32] recast difficult computational problems as games for humans to play. As an example, to date, computer vision systems have been poor at the general problem of identifying items in images drawn from a large class of objects. The ESP Game [31] asks two players who cannot otherwise communicate to try and guess matching words to describe the image. When the players agree, both players gain points, and the game has learned a label for the image. Friendsourcing extends the design principles of Games with a Purpose to address the challenges of collecting data from a small network. Games with a Purpose typically use points as motivators, randomly pair players to prevent cheating, and collect information that all players know but that computers do not know. Though friendsourced applications such as Collabio utilize game motivations such as point scores and leader boards, they lean just as heavily on social motivators such as social reciprocity, the practice of returning positive or negative actions in kind [10]. Rather than anonymously pairing random players to prevent cheating, we target users within established social groups to contribute data, relying on social accountability and profile management to discourage poor behavior [8]. Finally, rather than gather information common to all web-enabled humans, we directly target information that is known and verifiable only by a small social group: information about a friend [29].

Studies of contribution in online communities motivate several design decisions in friendsourcing. One danger is social loafing: users will exhibit little effort on a collective task when they believe that others will also contribute [17, 20]. Related to social loafing is diffusion of responsibility: when many individuals share the responsibility for an action that one person must perform, each individual feels less cognitive dissonance (i.e., guilt) for not acting [19]. However, individuals are likely to contribute to an online community when they are reminded of the uniqueness of their contributions, given specific, challenging goals, and helping groups similar to themselves [2, 23]. Thus, when friendsourcing, we challenge individuals’ (potentially obscure) knowledge of members of their own social group. Both active and loafing users can be motivated by comparing their activity to the median participation of the community [12], as in the kind of competition that Collabio and FeedMe have designed into their leaderboards.

System DesignsThis section outlines the systems that will comprise the core of my thesis. I have organized the systems around the three themes: outsourcing, friendsourcing, and data mining.

Outsourcing: Batched and RealtimePaid crowd workers are an excellent source of work when the designer cannot find an intrinsic motivator. They are also available in large number, and so are extremely helpful when prototyping. In this section, we describe an interface that embeds Amazon Mechanical Turk workers to support word processing tasks and outline a future realtime system.

SoylentWe have developed Soylent, a word processing interface that utilizes crowd contributions to aid complex writing tasks [6]. Soylent is people: its core algorithms involve calls to Mechanical Turk workers (Turkers). The system is comprised of three main components.

First, Shortn (Figure 1) is a text shortening service that cuts selected text down to 85% of its original length on average without changing the meaning of the text or introducing errors. The user selects the area of text that is too long, then presses the Shortn button to launch Mechanical Turk tasks in the background that identify overly wordy parts of the text and propose rewrites. Launching the Shortn dialog box (Figure 1), the user sees the original paragraph on the left. On the right, Shortn provides a single slider to adjust the length of the final paragraph. As the user drags the slider, Shortn computes the combination of crowd trimmings that most closely match the desired length. From the user’s point of view, as she moves the slider to make the paragraph shorter, sentences are slightly edited, combined and cut to match the desired length.

Second, Crowdproof (Figure 2) is a human-powered spelling and grammar checker that finds problems Word misses, explains the problems, and suggests fixes. Mechanical Turk workers read sections of text to find and fix errors that Word does not correct. It calls out edited sections with a purple dashed underline. If the user clicks on the error, a drop-down menu explains the problem and offers a list of alternatives.

Third, The Human Macro is an interface for offloading arbitrary word processing tasks. For example, we have used The Human Macro to format citations correctly and find figures for paragraphs. Launching the Human Macro opens a request form (Figure 3). The form dialog is split into two mirrored pieces: a task entry form on the left, and a preview of what the Turker will see on the right. The user then chooses how many separate Turkers he would like to complete the task, and whether the Turkers’ work should replace the existing text or annotate it with comments.

Figure 1. Shortn allows users to adjust the length of a paragraph via a slider. Red text indicates locations where cuts or rewrites have occurred. Tick marks represent possible lengths, and the blue background bounds the possible lengths.

Proposal: Realtime SupportSoylent can be slow: Turkers may take several minutes to complete the tasks after the user requests them. We believe that we can make crowd-powered interfaces even more engaging by eliminating this delay. So, our next step is a prototype system with real-time crowd-powered interaction. This system will pursue at least one of two mechanisms: 1) The Surgeon’s Assistant, placing a paid crowd worker in a position to aid the user directly, for example by screensharing with the user for a period of time; 2) The Quantum User, where crowd workers explore future possible states of the user interface and their actions are used to “backfill” the user’s task when the user arrives on a path that a Turker explored.

I propose to explore realtime support with a Photoshop plug-in. Photoshop is an extremely complex authoring system, and non-experts spend lots of time finessing the tools to do exactly what they want. Realtime support could significantly cut down on wasted time, and help automate complex image-editing workflows.

Technical ChallengesMechanical Turk costs money and it can be error-prone; to be worthwhile to the user, we must control costs and ensure correctness. We have developed a crowd programming design pattern called Find-Fix-Verify that splits complex crowd intelligence tasks into a series of generation and review stages [6]. These stages use independent agreement and voting to produce reliable results. Rather than ask a single crowd worker to read and edit an entire paragraph, for example, Find-Fix-Verify recruits one set of workers to find candidate areas for improvement, then collects a set of candidate improvements, and finally filters out incorrect candidates. This process prevents lazy crowd workers from contributing too little, or eager workers from contributing too much and introducing errors.

With respect to realtime crowd support, we expect to pursue technical means to lower latency and cost, for example multiplexing each Turker across several users at once.

Friendsourcing: Personalized InterfacesSocial data is critical to many interactive applications. For example, Yahoo! Answers cannot easily help with questions about the history of your small a cappella group or the way your friend delivered his marriage proposal. However, when information is known only to a small group in a social network, traditional crowdsourcing mechanisms struggle to motivate a large enough user population to share.

Figure 2. Crowdproof is a human-augmented proofreader. The drop-down explains the problem (blue title) and suggests fixes (gold selection).

Figure 3. The Human Macro is an end-user programming interface for automating document manipulations. Left: user’s authoring interface. Right: Turker’s preview.

In such situations, we bring social application design to bear via an approach we call friendsourcing [5]. Friendsourcing gathers social information in a social context: it lays out incentives for a user’s social network to share information or produce a desired outcome. Here, we friendsource for personalization, gathering descriptive information about a person that we can use to power interactive applications on their behalf. We have designed two systems to collect friendsourced data, as well as a series of interactive prototypes using the data.

CollabioWe developed Collabio (Collaborative Biography), a game that elicits descriptive tags for individuals within Facebook [4, 5]. Collabio (Figure 4) draws on information that your friends know about you. This information includes your personality, expertise, artistic and musical tastes, topics of importance, and quirky habits. The application leverages properties of the social network such as competition and social accountability to solve the tag motivation and accuracy problems within a social framework.

The main activity of Collabio is tagging friends. Players see a tag cloud that other friends have collectively authored by tagging the selected friend. When presenting this cloud, Collabio hides tags that the user has not guessed. Each hidden tag has its letters replaced with solid circles; for example, the tag THESIS appears as ●●●●●●. When a user guesses a tag that others also guessed, it is revealed within the cloud. For each guess, users receive points equal to the number of people who have applied a tag, including themselves. If they are the only person who guessed that tag, then they get 1 point; if there are 11 others, they get 12 points.

Collabio has so far gathered 7,780 unique tags on 3,831 individuals in 29,307 tagging events.

FeedMe Collabio encourages users to take on a new behavior – friend tagging – but we can do better by augmenting existing behaviors. Today, to find interesting web content, people rely on friends and colleagues to pass links along as they encounter them. In the FeedMe project [3], we study and augment

Figure 4. In Collabio, this user has guessed several tags for Greg Smith, including band, ohio and vegas.

link-sharing via e-mail, the most popular means of sharing web content today. We have developed FeedMe (Figure 5), a plug-in for Google Reader that makes directed sharing of content a more salient part of the user experience. FeedMe recommends friends who may be interested in seeing content that the user is viewing, based on what friends have shared with them in the past. It provides information on what the recipient has seen and how many emails they have received recently, and gives recipients the opportunity to provide lightweight feedback when they appreciate shared content. FeedMe also introduces a novel design space of mixed-initiative social recommenders: friends of the user vet the material on the user’s behalf.

By making sharing easier, FeedMe can implicitly learn user models – it simply tracks which articles are sent to which friend. It utilizes a user’s social network to produce accurate user models without the user’s involvement.

Applications of FriendsourcingFriendsourced data can open up new avenues for interactive applications. We have produced a pair of novel crowd-powered interfaces using our friendsourced data: a tag cloud aggregator for tag visualization and exploration and an expert-finding question answering system.

Collabio Clouds (Figure 6) aggregates tag clouds from Collabio based on user queries [5]. The user can query his or her own tag cloud as well as the aggregated tag cloud of friends, all Collabio users, users with specific other tags (like tennis or Adobe), or users in Facebook networks or groups. Collabio Clouds allows users to explore questions such as: What do the tag clouds of members of the Penn State network look like? What other tags show up on individuals tagged with machine learning?

Figure 7. Friendsourced QnA is a question and answer system that uses friendsourced data to find friends and friends-of-friends who can answer your questions.

Figure 6. Collabio Clouds comparing users tagged with washington to users tagged with georgia tech.

Figure 5. The FeedMe plug-in for Google Reader suggests friends, family, and colleagues who might be interested in seeing the post that you are reading.

The second application is an expert-finding system [5]. Question and answer (QnA) systems such as Yahoo! Answers rely on a large community of answerers actively seeking out questions. Friendsourced data opens up new possibilities: a QnA system can now route questions only relevant within the user’s social network, such as When is the next HCI group meeting?, or Who might be interested in starting an IM football team at Google? Users ask questions, and Friendsourced QnA (Figure 7) searches over the user models to identify friends and friends-of-friends who are most likely to be able to answer the question.

Data Mining: Remixing ExpertsWe know that we can incentivize a social group to share information that will power an interface, and that (by using payment as an incentive) we can embed live humans in interfaces to produce new

Figure 8. PingPong++ is a ping pong table that records the location of every ball hit, and projects visualizations (like this koi pond) back onto the table..

Figure 9. PingPong++ mines the recorded games to recommend actions to the player. Left to right: a heatmap of where your hit might be returned, an animating circle showing where and when players would hit the ball when in the player’s situation, and arrows showing what exper players would do in the current situation.

interactive possibilities. However, these interfaces require the crowd to do up-front work, and that work is only useful to one user.

As a first exploration of this space, we have developed PingPong++, a crowd-powered ping pong table (Figure 8). PingPong++ senses the position of every ball hit on its surface, and saves the games to a database. It then mines that database to recommend actions to the player (Figure 9). For example, with the Arrows visualization, the table draws an arrow from where the ball landed to the place an expert player would return the hit. We gathered expert player data by recording games played on the table by the best players in a local ping pong ladder.

In the final piece of this thesis, I propose interaction possibilities that arise by aggregating user traces. In particular, usage patterns of expert users can be leveraged to aid the average user. The result of this project will be a system that allows users to seamlessly stitch together expert-created data to create output that they would not otherwise have been able to author. This work builds on previous research which aggregates activity traces or datasets to produce novel interfaces (e.g., [7, 13, 26]).

Critically, this research still needs to identify a task to support. One hypothetical system is a smart camera trained on top photos from Flickr. As the user composes a shot, the camera compares the photo to a database of quality photos in terms of light, color, orientation, subject placement, and other machine-viewable features. The camera can then recommend actions like zooming in, moving the photograph subject, or adjusting the room lighting. The user can interactively adjust the shot as the camera provides feedback. This domain will likely change, but the general research idea will be similar.

Status and TimelineOf the systems described in this paper, FeedMe, Collabio, and Soylent have been published. The remaining two – the realtime outsourced support system (The Surgeon’s Assistant) and the expert aggregation interface – are still in early stages of design. We hope that participation in the doctoral colloquium will help us refine the map of uncharted design space this thesis purports to explore and will provide feedback on the eventual designs of these prototype systems.

Of the six systems I plan to discuss in my thesis, four of them are complete. Soylent, Collabio and FeedMe have been built and accepted for publication. PingPong++ is under review. The real-time follow on to Soylent and the crowd-powered remixing project are not yet complete. Other projects, including Eddi (friendsourcing/data mining) and Sociapedia/stat.us (friendsourcing) will be woven into the thesis as appropriate, but may not constitute major sections. My current plan is to follow this timeline:

September 2008. Complete Collabio and submit it for publication. complete – Collabio rejected from CHI 2009, published in UIST 2009 and friendsourcing follow-up published in TOCHI

September 2009. Complete FeedMe and submit it for publication. Complete Eddi and submit it for publication. complete; FeedMe accepted to CHI 2010, Eddi rejected from CHI 2010 and published in UIST 2010

March 2010. Complete Soylent and submit it for publication. complete; Soylent published in UIST 2010.

August 2010. Complete PingPong++ and submit it for publication. complete; PingPong++ submitted to CSCW 2010

September 2010, backup March 2011. Complete the real-time Soylent follow-on and submit it for publication.in progress; most likely for UIST 2011 at this point

March 2011, backup September 2011. Complete the crowd-powered remixing project and submit it for publication.

Spring 2012: Job search. June 2012. Graduate.

The timing is frontloaded because my final year in graduate school does not provide many opportunities for publication that would impact my job search. UIST 2011 is my last feasible publication venue that would publish before I need to submit my job search materials. Additionally, I’d like to give myself an average of 2 publication cycles to get a paper accepted.

ContributionsThis thesis will demonstrate that social data from an online database, a social network site, and a labor market can each be used to produce user interfaces that are more intuitive and powerful than the ones we use today. We believe that crowd-powered interfaces can support high-level tasks like personalization and writing by substituting human cognition where algorithms and interfaces cannot succeed yet. We will make contributions in three areas: 1) novel end-user interactive applications, 2) new human computation algorithms to manage contributors, and 3) designs of social systems that incentivize group members to contribute desired information.

ConclusionWe pursue the design of crowd-powered interfaces: interfaces that integrate human activity and cognition to support complex end-user tasks. Data and algorithms for these interfaces may be outsourced to a labor market like Mechanical Turk, friendsourced to the user’s social network, or mined from usage patterns of Internet users. We demonstrate prototype systems for each of these application areas to support needs that AIs and user interface design have not yet successfully addressed.

References

1. Ackerman, M.S. and Malone, T.W. Answer Garden: a tool for growing organizational memory. GROUP '90, ACM Press (1990), 31–39.

2. Beenen, G., Ling, K., Wang, X., et al. Using social psychology to motivate contributions to online communities. CSCW '04, ACM Press (2004), 212–221.

3. Bernstein, M., Marcus, A., Karger, D.R., and Miller, R.C. Enhancing Directed Content Sharing on the Web. CHI '10, ACM Press (2010).

4. Bernstein, M., Tan, D., Smith, G., et al. Collabio: A Game for Annotating People within Social Networks. UIST '09, ACM Press (2009).

5. Bernstein, M., Tan, D., Smith, G., et al. Personalization via Friendsourcing. ACM Transactions on Human-Computer Interaction (TOCHI) 17, 2 (2010), 1-28.

6. Bernstein, M.S., Little, G., Miller, R.C., et al. Soylent: A Word Processor with a Crowd Inside. UIST '10, ACM Press (2010).

7. Chen, T., Cheng, M., Tan, P., et al. Sketch2Photo. ACM Transactions on Graphics 28, 5 (2009).8. DiMicco, J.M. and Millen, D.R. Identity management: multiple presentations of self in facebook. GROUP '07,

ACM Press (2007), 383–386.9. Gamon, M., Basu, S., Belenko, D., et al. Blews: Using blogs to provide context for news articles. Proc. ICWSM

'08, (2008), 60-67.10. Gouldner, A.W. The Norm of Reciprocity: A Preliminary Statement. American Sociological Review 25, 2 (1960),

161–178.11. Gutwin, C. and Greenberg, S. A Descriptive Framework of Workspace Awareness for Real-Time Groupware.

Computer Supported Cooperative Work 11, 3 (2002), 411–446.12. Harper, F.M., Xin Li, S., Chen, Y., and Konstan, J.A. Social Comparisons to Motivate Contributions to an Online

Community. In Persuasive Technology. Springer Berlin Heidelberg, Berlin, 2007, 148–159.13. Hartmann, B., MacDougall, D., Brandt, J., and Klemmer, S. What Would Other Programmers Do? Suggesting

Solutions to Error Messages. CHI '10, ACM Press (2010).14. Heer, J. and Bostock, M. Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization

Design. CHI '10, ACM Press (2010).15. Heer, J., Viégas, F.B., and Wattenberg, M. Voyagers and voyeurs: supporting asynchronous collaborative

information visualization. Conference on Human Factors in Computing Systems, (2007), 1029.16. Kammerer, Y., Nairn, R., Pirolli, P., and Chi, E.H. Signpost from the masses: learning effects in an exploratory

social tag search browser. CHI '09, ACM Press (2009), 625–634.17. Karau, S. and Williams, K. Social loafing: A meta-analytic review and theoretical integration. Journal of

Personality and Social Psychology 65, 4 (1993), 681–706.18. Kittur, A., Chi, E.H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. CHI '08, ACM Press (2008).19. Latané, B. and Darley, J.M. The Unresponsive Bystander: Why doesn't he help? Appleton-Century Crofts, New

York, 1970.20. Latané, B., Williams, K., and Harkins, S. Many hands make light the work: The causes and consequences of

social loafing. Journal of Personality and Social Psychology 37, 6 (1979), 822–832.21. Little, G., Chilton, L., Goldman, M., and Miller, R.C. TurKit: Human Computation Algorithms on Mechanical Turk.

UIST '10, ACM Press (2010).22. Quinn, A.J. and Bederson, B.B. A Taxonomy of Distributed Human Computation. 23. Rashid, A.M., Ling, K., Tassone, R.D., et al. Motivating participation by displaying the value of contribution. CHI

'06, ACM Press (2006), 955–958.

24. Ross, J., Irani, L., Silberman, M.S., et al. Who Are the Crowdworkers? Shifting Demographics in Amazon Mechanical Turk. alt.chi '10, ACM Press.

25. Sala, M., Partridge, K., Jacobson, L., and Begole, J. An Exploration into Activity-Informed Physical Advertising Using PEST. Pervasive '07, Springer Berlin Heidelberg (2007).

26. Simon, I., Morris, D., and Basu, S. MySong: automatic accompaniment generation for vocal melodies. Proc. CHI '08, ACM Press (2008).

27. Snow, R., O'Connor, B., Jurafsky, D., and Ng, A.Y. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. ACL '08, (2008).

28. Sorokin, A. and Forsyth, D. Utility data annotation with Amazon Mechanical Turk. CVPR '08, (2008).29. Toomim, M., Zhang, X., Fogarty, J., and Landay, J.A. Access control by testing for shared knowledge. CHI '08,

(2008), 193–196.30. Viegas, F.B., Wattenberg, M., van Ham, F., et al. ManyEyes: a site for visualization at internet scale. IEEE

Transactions on visualization and computer graphics 13, 6 (2007), 1121–1128.31. von Ahn, L. and Dabbish, L. Labeling images with a computer game. CHI '04, ACM Press (2004).32. von Ahn, L. Games with a Purpose. Computer 39, 6 (2006), 92–94.33. von Ahn, L. Human computation. International Conference On Knowledge Capture, (2007).