Real-time Search and Discovery of the Social Web

8/14/2019 Real-time Search and Discovery of the Social Web

http://slidepdf.com/reader/full/real-time-search-and-discovery-of-the-social-web 1/20

© 2009, Spark Media Solutions, LLC.http://sparkmediasolutions.com/ Page 1

Real-Time Search andDiscovery of the Social Web

By David SparkFounder, Spark Media SolutionsDecember 7th, 2009 Twitter: @dspark

Updated: December 10th, 2009

http://twitter.com/dspark

http://twitter.com/dspark




Table of contents

Executive summary

Real-time interaction on the InternetTraditional search engines and the real-time webPublishing for a few, implications for allThe Michael Jackson wake-up callLooking for real-time answersWhat real-time search delivers

The technologies and behaviors behind real-time searchThe APIsThe protocolsThe function of a real-time search engine

Market dynamics and ecosystemIndexing real-time data: explicit vs. implicitTwitter ecosystemTraditional search playersReal-time search startupsWeb users rely on each other for a successful real-time web

Industry economics and business models

Emerging business models for real-time search

What should we expect from real-time search

Endnotes

Author

Real-Time Search and Discovery of the Social Web

3

4

9

11

16

18

19

20




Executive summary

Each day, users of the social web post more than230 million pieces of content. They’re sharingall types of published media — videos, pictures,tweets, Facebook status updates, blog posts,comments, reviews, and more — with theirimmediate community and the greater social web.Given the short lifespan of much of this information,traditional search engines have historically ignoredpublishing a significant portion of this content. Thisgap in search engine reporting has opened up newopportunities for real-time search and discoveryengines. On a minute-by-minute or even second-by-second basis, these new search engines digestand make sense of what people are talking aboutacross the social web. But their autonomy fromthe traditional search engine players is aboutto dissipate. The real-time search landscape ischanging as both Google and Microsoft’s Bing formreal-time search partnerships, expanding theirdefinition of searchable content to include thepublication of dynamic social web content.


NOTE: This report focuses mainly on user-generated real-time content. It purposely omitsmachine-to-machine real-time content and longexplanations of location-based information.

People’s use of search has evolved with the growthof social media. It’s estimated that as much as 40percent of all searches have a real-time component.That means new competitors who aim to servethis market by providing answers specifically forreal-time queries could take away up to $40 billiondollars from traditional search engines. That’s

just the beginning revenue potential as real-time search can deliver additional money makingservices such as real-time conversations, discovery,programming, and trend tracking.

The future of the real-time search market is beingshaped by emerging technologies, new players,the evolving incumbent search players, and thechanging ways web users interact with each other.



Page 4

Real-time interaction on the Internet

Google has proven to everyone that there’s moneyin search. Competitors Microsoft Bing, Yahoo (nowpartnered with Microsoft) and Ask have all madestrong efforts to steal even the smallest marketshare away from Google. The fight has been worthit because it’s estimated that capturing a singlepercent of the search market can increase marketcapitalization by $1 billioni. For years they’ve beenfighting over the same pie of long-term indexed

content. They haven’t been paying attention to theever widening and shifting pie that is the real-timeweb.

Traditional search’s foundation is built ondurability over time. Google and its traditionalsearch competitors operate like a well-cataloguedencyclopedia, indexing data and its popularity overtime to find content which proves to be the mostvaluable. Only information that’s been culled bya community of other long-term web publishersappears at the top of search engine results.However, much of the content being created via

social media tools such as Facebook and Twitter isbeing created for a specific moment, rather thanits long-term archival value. Status updates andtweets are published to be consumed as they’rebeing created. A single tweet’s value can come andgo in less than a minute.


Traditional search engines and the real-time web

Therefore, real-time web content’s lifespanis antithetical to a traditional search engine’salgorithms. Since content doesn’t appear or getreferenced in the ways traditional search enginesdiscover and evaluate content, it has largely beenignored. Prior to its deal with Twitter, Google saidit had been indexing real-time content such asTwitter for years. Yet it also admits it returns veryfew results from Twitter and doesn’t offer the same

experience as real-time search engines do.

© 2009, Spark Media Solutions, LLC.http://sparkmediasolutions.com/



Page 5


Each day the real-time web generates more than230 million pieces of contentii. That includeseverything from tweets, Facebook status updates,bookmarks, article recommendations, blog posts,comments, videos, pictures, and more. Most of thatcontent is being produced for an already connectedsocial networking audience such as online friends,followers, or fans.

While real-time content may initially be createdfor an existing audience, when it’s aggregated andanalyzed, it has larger implications for everyone.Real-time search engines can uncover revealingin-the-moment trends, watch stories unfold, andconnect users across the world as they experienceand comment on a single event.

Real-time web content is being created by usersthat were traditionally seen only as searchconsumers, but their reactions, now published,have value. Any topic or issue that is evolving (orotherwise timely) benefits from real-time search.

Examples include:

• Concerned citizens tracking tragedies or politicalunrest, such as the Iran election protests.

• MTV fans collectively commenting on KanyeWest’s interruption at the MTV Video Awards.

• Obsessive football fans getting up-to-the-minuteresults on their fantasy teams.

• Companies providing customer service at themoment people express interest or frustration

with their product.


Publishing for a few, implications for all

During any hot news story, consumers haverepeatedly proven their hunger for up-to-dateinformation. That behavior has been expressedrepeatedly through real-time web searches. A singleuser can enter the same search query as manyas ten times a day, according to real-time searchsources.

Our behavior is changing as we’re seeing what the

real-time web can provide. People are getting usedto watching stories unfold on Twitter. “In the ageof real time, Twitter is Walter Cronkite,” arguedTechcrunch. News organizations are even using thereal-time web as a resource to track and sourceinformation. Cable news networks are constantlystruggling to find stories and images to fill uphours of programming. Especially during a crisis,viewers take comfort in the constant availabilityof information, sometimes to the sacrifice of itsquality. “Emotions are high during a crisis andpeople find comfort in information — regardless of the quality of that information,” said Keri Stephens,

assistant professor of communication studies at TheUniversity of Texas at Austin.




Page 6


During live events with unpredictable storiesthat unfold over time — disasters and sportingevents, for example — the demand for real-timeinformation skyrockets. People are eager for up todate information. The most notorious example wasthe news of Michael Jackson’s death in June, 2009.

As the story broke, there were so many queries for “Michael Jackson” that for a full 25 minutes Google

News mistook the surge for an automated attack and some people received a “We’re Sorry” page fortheir results. It is believed that Google’s responsewas the result of an automatic trigger set to go off when there’s such a dramatic surge in searcheson a single term. While some people may haveseen the TMZ or Perez Hilton articles that brokethe story, Google failed to reveal the public chatterusers were looking for. Google News indexes aselect number of sources which includes TMZ, butnot the entire blogosphere. Still, even during thisdramatic episode, whether they caught the newsstories or not, many saw at the top of the page, the

same two results as they do today:michaeljackson.com and Jackson’s Wikipedia page.


The Michael Jackson wake-up call

As bits of information on the real-time webappeared, the story of Michael Jackson’s deathstarted to unfold, most notably on blogs and viaTwitter and Facebook status updates. The storywasn’t being explained in a pre-digested methodicalstandard published format, but rather via astream of community published and distributedinformation. Over time, the story revealed itself as real-time web users shared and pointed to

information gleaned from other real-time webusers.

Many web users believe that Google can providethe best information for search needs, but whenit comes to real-time information generated anddistributed via the social web, that assumptionwas wrong. The response to Jackson’s deathdemonstrated that by ignoring the real-time web,Google and its search engine competitors wereproviding a poor experience to a real-time query.


http://googleblog.blogspot.com/2009/06/outpouring-of-searches-for-late-michael.html







Page 7


Those users conducting searches for personalitiescurrently in the news are inevitably trying to findthe latest information. Wanting real-time resultsis a common search behavior, and it comprisesan estimated 15-40 percent of all searchesiii. Inaddition to consumer interest in trends, breakingnews and the latest meme, businesses are usingreal-time search engines, also sometimes knownas social media monitoring tools. They’re looking

for greater insight into public sentiment about theirproducts and operations and want to be alerted torelevant industry conversations.

Google has prided itself on incorporating differentkinds of media into its search results, as it hasstarted to fold video, image, and blog searches intoits overall universal search. But when it came tothe real-time web, Google, Yahoo, and MicrosoftBing were all struggling to satisfy users’ demand forreal-time user generated information.


Looking for real-time answers

As of October 2009, however, both Google andMicrosoft have made search deals with Twitter.Microsoft also has a search deal with Facebook, inwhich it is an investor. The details of these searchdeals are becoming evident now. On the date thisreport was published, Google announced its rolloutof real-time search incorporated into general searchwith results from social services such as Twitter,blogs, MySpace, FriendFeed, and even Facebook,

Techcrunch reported.

Even with this announcement, two of the biggestreal-time content providers, Twitter and Facebook,only comprise a small portion of the real-timecontent being shared on the social web. Accordingto social media data shared via the AddToAny widget, Twitter only comprises 10.8 percent of theuser-generated real-time web and Facebook 24percentiv, indicating that there’s still plenty morereal-time content to index.


http://addtoany.com/

http://addtoany.com/



Page 8


Real-time search engines have multiple ways of collecting and indexing data as well as differentapproaches to displaying results, which oftenrepresent one or a combination of the following twomethods:

1. The fire hose - Results just stream in as thecontent is being produced and indexed from aparticular source or set of sources.

2. Relevancy now - When multiple people arecurrently linking to a published story, video,or other, indexing this sharing via social toolssuch as Twitter and Digg can show currentand collective interest. Relevancy gives rise todiscovery, allowing searchers to see what’s hotnow whether universally, within a specific locale,or within a specific topic.


What real-time search delivers

Traditional search engines are most concerned withrelevance over time, but real-time search enginesmust factor in both recency and popularity. Findingthe appropriate mix to calculate spikes and changesin public interest of terms, topics, and stories is anongoing effort for real-time search engines.




Page 9

The technologies and behaviors behind real-time search

One of the initial core issues for traditional searchengines has been the time it takes to index newlypublished information. Ten years ago, a recentlypublished piece of web content could take weeksto get indexed. The problem stemmed from theconstant need to poll all web sites to see if therewas any new information to index and wait for aresponse. This need for two-way communicationfor site updates, even if nothing had changed, was

horribly inef ficient.

About a year ago, social media sites realized that,given the volume of content being created, theywould have to gather this information more quicklythan with traditional polling. They began usinglong polling which still sent out a request, but thereceiver would just sit and wait and only respond if content had changed, thereby eliminating “we havenothing” responses. This dramatically increased thespeed with which content could be indexed.


Today, real-time search tools are based on utilizinga variety of different APIs (application programinterfaces) and technologies that allow them togather information very rapidly, sometimes as fastas two to three times a second. Below is a review ofthe technologies, and how they’re being used.

Twitter Search API — Of all the real-time searchtools, Twitter has the best combination of valuablereal-time dialogue, open access to indexing, and acritical mass of usage. This entire discussion of real-time search can be traced back to the introductionof Summize, an early popular real-time Twittersearch engine that was so good that Twitter boughtit and renamed it as Twitter Search.

Even though, by one estimation, Twitter comprisesless than 11 percent of the user-generated real-time web, indexing the tweets and the sites theypoint to can provide seemingly accurate real-timesearch results. Some real-time search enginesare fully based on this API and just index Twitterconversations.

Facebook’s AP I — Creating similar and moredynamic content than Twitter, Facebook has beendeveloping its own real-time search engine and willsoon be offering results for Microsoft Bing. Afterthat, Facebook’s API is closed for outside searchengines. Facebook application developers canscan user content only if each individual Facebookuser approves it. In addition, not all of Facebook’scontent is open, even within Facebook. Users have

control over what content they want indexed andviewable by the public and their friends.

The APIs




Page 10

The technologies and behaviors behind real-time search

Simple Update Protocol (SUP) - A “pingfeed” that any web service, usually some type of publishing application, can use to alert subscribersof a feed update. It was developed by Friendfeed,a service that alerts subscribers on many differentkinds of content feeds. Simple and compact, SUPeliminates frequent polling, thereby making it farmore ef ficient.

PubSubHubBub and RSSCloud - Bloggingprotocols for distributing content in real time. Thefirst, developed by Google, is an extension of theATOM syndication feed. The second is an elementof RSS 2.0 and it’s being used by WordPress.com.Rather than sending updates directly from publisherto subscriber, these protocols leverage a “hub” thatmanages the complexities of distribution, taking theworkload away from the publishers. This can cut thetime needed to deliver content from blog publisherto RSS reader to just seconds, from what usedto take 15-60 minutes using a direct publisher tosubscriber interface.


Extensible Messaging and Presence Protocol(XMPP) - This is not a feed managementprotocol, but rather a 10-year-old series of openXML technologies for presence and real-timecommunication, such as instant messaging, VoIP,machine-to-machine communications, and nowsocial networking. Some real-time search engines,such as Collecta, are utilizing the historical breadthof real-time communications tools built on top of

XMPP to create their real-time search engines.

Real-time search engines perform the followingfunctions:

Surfacing - As the seemingly endless stream of data comes in, the real-time search engine indexes

and finds trends, bringing them to our attention,dynamically. This is sometimes not a function of

“search” but rather of “discovery.”

Rank real-time results - Algorithm that integratesa variety of factors: freshness of content, authorityof domain that published content, authorityof people recommending the content, and theincreasing volume of people recommending the

content.

Manage volume and spam - Not only do real-time search engines need to collect information inreal time, they also need to get rid of garbage inreal time.

The function of a real-time search engine

The protocols




Page 11

Market dynamics and ecosystem

The major gatekeepers of the explicit real-time webare currently Twitter, which is open, and Facebook,which is closed. The explicit real-time web refersto content that users choose to publish publicly orfor their select community. Content can be either aself-contained original statement or it can point tosome other web content (e.g. “check out this blogpost http://tinyurl.com/...”).

The implicit web refers to what people don’texpressly publish, yet is very measurable behavior.Traditional search engines index the implicit web,monitoring user activity such as what users searchfor and what links they click.

The overwhelming majority of real-time searchengines only index the explicit web. This approachhas its limitations. In particular, the only contentthat is seen and relevant is what people publishand others choose to share. Harvard BusinessReview estimates that 90 percent of the real-timeweb is being published by just 10 percent of usersv.

On Twitter, only 5 percent of the users create 74percent of the contentvi. Depending how you lookat it, those sub-10 percent users are either arepresentative sample or highly skewing the results.

The only way to gauge broader user activity andinterest is to measure online behavior, or theimplicit web. Searching the implicit web requirestons of data, and Twitter search is probably the onlyreal-time search engine that’s got enough activityto track passive behavior, currently. However,Twitter Search doesn’t seem to be factoring anyimplicit behavior into its results.


Conversely, OneRiot, which doesn’t get nearly asmany queries as Twitter search, is still indexingimplicit behavior through search partnershipsand offering a free toolbar. They currently havepartnerships with 80 search engines, such as Zibb and Scour, plus users can install a toolbar thatmonitors their online behavior anonymously, and inreturn they get better real-time insights into theirinterests.

Using its search engine, its toolbar, and data fromits partners’ search engines, OneRiot tracks thelinks users click on and their “dwell time.” Forexample, if a user clicks on a link and immediatelybounces back, spending no time, then that’s asignal that the link may not be valid. Conversely,if a user clicks on a six-minute video and theywatch for five and a half minutes, then it’s probablyvaluable.

OneRiot’s search partner network has increased itsbranding and opened up the number of queries it

is able to ingest and analyze — from 5 million inJune without search partners to now more than 40million monthly searches across its entire partnernetwork, which includes the 3 million registeredopt-in users of its toolbar.

For implicit search to be valuable, you needfeedback. More importantly, you need a lot of feedback. Achieving that critical mass was onlypossible when OneRiot syndicated its search to itspartner network thus enabling them to improvethe quality of search results more quickly. Explicitsearch alone doesn’t give you a full picture. In

addition, their ad inventory jumped dramaticallywhich allowed them to work with large advertisersand that resulted in increased revenue.

(cont’d)

Indexing real-time data: explicit vs. implicit


http://zibb.com/

http://scour.com/

http://scour.com/

http://scour.com/

http://zibb.com/



Page 12


Newly launched real-time search engine Wowd also has an elective add-on so it can track implicitsearch behavior through its own peer-to-peernetwork. However, given Wowd’s limited audienceand lack of a search partner network, it simplydoesn’t have enough traf fic to create valuableimplicit real-time search data. In initial tests, headto head, OneRiot’s results are more timely andaccurate than Wowd.


Adding implicit behavior is necessary for valuablereal-time search results as it can show interest inthe overwhelming cases when people choose not topublish their interest over the real-time web.

Indexing real-time data: explicit vs. implicit (cont’d)

For years, “YouTube” was synonymous with “onlinevideo.” Three years ago if you asked someonewhere they watch video online, they’d probably sayYouTube. Today, YouTube is still dominant, but it’s

got well over 1,000 competitors, and it no longerowns the term “online video.” Ironically, a searchfor “video” or “online video” on Google (YouTube’sowner), places YouTube in the middle of the firstpage of results. Other brands, such as Hulu andMetacafe, have emerged, taking a bite out of YouTube’s online video dominance.

Like YouTube did three years ago, “Twitter” iscurrently enjoying that public mindshare of beingsynonymous with “real-time web.” Facebook hasplenty of real-time conversation as well, but other

than itself and Microsoft Bing it’s not open for fullpublic digestion like Twitter.

By opening its data through an open API, Twitterhas become a catalyst for the real-time web. Itscontent and intellectual property is making thisdiscussion possible. An entire ecosystem of Twitterapplications and search engines have been built

completely using the Twitter APIs. Many haveargued that Twitter could make money if it licensedits API. If Twitter did charge for its API, however,the sizeable ecosystem of applications using it

probably wouldn’t have developed.

Most real-time search competitors are simplydigesting mostly Twitter data in easy to consumechunks, or making the content more discoverable.There are two standouts, however. OneRiot andCollecta have their own intellectual propertyand established relationships that position themahead of other players in the market. OneRiot hasrelationships with browsers to distribute its toolbar,relationships with other search engines, and it’smeasuring both explicit and implicit real-time

behavior. Collecta’s engine is using the real-timecommunications tool XMPP to constantly retrievesearch queries and present them in a very fastviewable stream of real-time programming. Its truemarket value is yet to be seen. That’s why Collectahas opened up its API up to developers in hopesthat one will create a killer application to raiseawareness of the value of its search data.

Twitter ecosystem


http://wowd.com/

http://wowd.com/



Page 13


With the introduction in August 2009 of a newindexing methodology, code named “Caffeine,” Google has dramatically increased the speed atwhich it indexes web content. There’s now an optionto see latest results in the past hour. Unsanctionedby Google, an outside developer introduced ahack that can show search results within the pastsecond. Such results indicated that Google waspoised to make its own play in the real-time web,

beyond its partnership deal with Twitter.


As mentioned earlier, Microsoft’s Bing haspartnership deals with Facebook and Twitter, and atthe time of writing this report, rumors have startedcircling that Yahoo will form a partnership deal withOneRiot.

These initial real-time search partnerships appear tobe Band-Aid solutions, however, aimed at stemmingthe flow of people from traditional search engines to

real-time search engines. In early implementations,we’re seeing real-time results branded within thedisplay of the traditional search engine’s results.This is unlike all previous specialized searchintegration, which have been integrated into overallsearch results.

Traditional search players

Real-time search is a true competitor to incumbent

search players, in that it can answer questionstraditional search algorithms cannot. Because of incidents like Michael Jackson’s death, traditionalsearch engines have realized that they must beginreturning real-time results or they will lose traf fic,and subsequently revenue, to real-time searchengines.

Like specialized searches that have come before(e.g. video, pictures, blogs), custom searchcompanies have been the first to tackle the uniquesearch problem. There are dozens of real-timesearch engines that simply index Twitter content.

Below, we highlight select companies that eitherhave a significant presence in real-time search orare providing a unique service.

almost.at - Focused on tracking discussion in real

time around specific live events. Tracks tweets,tweets linking to blog posts, and photos. Unique

feature is it records live discussions like a DVR.Want to know what everyone was saying exactly athalf-time? You can “rewind” to that point and see.

TweetMeme - Excellent branding as it’s slowlybecoming the ubiquitous tool to use on blogsposts. Theory is more tweets for a single piece of content indicates more people are interested in it.A single click of their retweet button automaticallylaunches Twitter with the content of the post and ashortened URL already filled in. Search engine only

looks at time stamps and number of times a specificarticle or post is retweeted. Each retweet is equallyweighted.

(cont’d)

Real-time search startups


http://zedomax.net/2009/09/15/google-hack-how-to-get-real-time-results/

http://almost.at/

http://tweetmeme.com/

http://tweetmeme.com/

http://almost.at/

http://zedomax.net/2009/09/15/google-hack-how-to-get-real-time-results/



Page 14


OneRiot - Combines both explicit (what peopleare posting) and implicit (what people click on andconsume) real-time web behavior. Also trackingDigg, Delicious, and YouTube. Branded algorithmPulseRank is a real-time version of Google’sPageRank.

Topsy - Indexes only Twitter. Ranks results bytimestamp and tweeter’s influence. For example,

a single retweet from Digg founder and populartweeter Kevin Rose is worth 341 retweets from theleast influential tweeter, said CEO Vipul Prakash.

Twingly - About.com for the real-time web. Userspick a topic they want to curate and create theongoing searches on that topic. Stories are revealedbased on those searches and people can addcomments to the stories.

Twendz - Fire hose results of Twitter search,but attempts to show sentiment (via color codes)and reveals trending subtopics to your search.

Information is constantly being refreshed andupdated.

YourVersion - A discovery engine based onindividuals’ unique interests. Set up a profile withall collected interests and YourVersion deliversupdates. Very impressive UI makes it easy tonavigate and share.

Collecta - Combining real-time search withreal-time communications to deliver a constantstream of fast results. Appears like a channelof programming. No aggregation or digesting.

Saves queries allowing users to “change channels” between streams of live web programming.


Scoopler - A hybrid between Collecta and OneRiot.Offers a channel of streaming programming, but notas fast. Delayed 45 seconds to one minute. Scansmultiple additional feeds such as YouTube, blogs,Twitpic, Digg, Delicious, and Flickr. Aggregatespopular results and includes discovery bymaintaining a top ten list of hot topics.

CrowdEye - A Twitter-only aggregator. Search

results provide related terms allowing for deeperdiscovery on search query. Simple and niceinterface for search and discovery of hot newsstories.

Twazzup - Very similar to CrowdEye and Scoopler.Twitter-only aggregator with an updated stream of content. Provides not only related terms but alsotweeters that are active on the subject.

Surchur - A discovery engine that tracks theexisting trend reporting on Twitter, Google, Yahoo,CNN, Technorati, and Bing.

Insttant - Visually stunning real-time searchaggregator. Only indexing Twitter, this programcreates a snapshot view of online discussion andtries to determine a sentiment score. Are peopletalking positively or negatively about the subject?Sentiment engine is far from perfect admits CEOJoe Langevin, as it is dif ficult to identify sarcasm.Insttant has plans for a freemium business model.They’ll offer the attractive graphical snapshot forfree and then charge for detailed analytics.

(cont’d)

Real-time search startups (cont’d)


http://oneriot.com/

http://topsy.com/

http://twingly.com/

http://twendz.com/

http://yourversion.com/

http://collecta.com/

http://scoopler.com/

http://crowdeye.com/

http://twazzup.com/

http://surchur.com/

http://insttant.com/

http://collecta.com/

http://yourversion.com/

http://insttant.com/

http://surchur.com/

http://twendz.com/

http://twazzup.com/

http://twingly.com/

http://crowdeye.com/

http://topsy.com/

http://scoopler.com/

http://oneriot.com/



Page 15


We’ll definitely see some level of consolidationamong real-time search engines as well as somekey acquisitions by larger players. Real-timesearch startups best hope is to impress the searchincumbents with their offerings whether it bebreadth of indexing, quality of results, usage base,usability, or new discoverable content.

Impressing Google and Microsoft may not be

strictly necessary as real-time search deliversa variety of different experiences not normallyseen in traditional search. Most notably thosenew experiences include discovery and a channelof programming content that will be bothcompetitive and compatible with traditional media


programming. Like TV, radio, print, and online,real-time web programming can be produced(a.k.a. curated) and provide a source of news andentertainment for which advertising can be soldagainst it.

While it’s estimated that real-time search couldconstitute as much as $40 billion of the traditionalsearch market, it could also represent a lot

more new revenue, as real-time search opensup new avenues for search behavior and relatedmonetization that haven’t been appropriate for thetraditional search market.

Real-time search startups (cont’d)

The real-time web is more than just content to beindexed and searched against. The content that’sbeing provided is revealing sentiments, ideas, andopinions of anyone connected to the Internet. Withreal-time search, the traditional behavior of usinga search engine to find what you want then leavingis changing. We’re seeing the following searchbehavior changes with real-time search.

Real-time search is becoming a programmingchannel — Initially, real time search has been an

auxiliary channel adding more value to an existingexperience (e.g. “I’m watching ‘American Idol’ rightnow, what are other people saying about it?”). Morerecently, it’s become a primary channel of content,especially at conferences where there is no otherprogramming (e.g. a TV show) and only thoseattending can create the content.

Real-time searchers are entering the samequery multiple times a day - This behaviordoesn’t usually happen with traditional searchengines. A repeated query shows a tendency towant to track certain subjects. Recognizing thisbehavior, Collecta saves searches so users canclick and change channels between their queries,allowing them to switch back and forth to seewhat’s changed.

Because of these emerging behaviors, real-

time search will affect many industries includingmajor brands, live events such as sports andentertainment, news, and marketing. Real-time search affects many industries beyond justtraditional search.

Web users rely on each other for a successful real-time web




Page 16


Expect to see several new business models evolvefrom real-time search. Some are already in theirnascent stages of development.

Paid search – Advertising is the core revenuesource for traditional search engines. Formarketers, real-time search adds another levelof relevancy, which is time, and in some cases,location. It’s an even more qualified version of

search engine marketing (SEM). If users aresearching multiple times a day on the same query,advertisers will have many times in a given day tomarket to them. They can use that knowledge tobuild a story and alter their messaging throughoutthe day.

Licensing API s - A possible argument for howTwitter could make money. Right now its API isfree. Could that possibly move into a paid model?Many technologies may lose audience interest if they force payments on developers. Some searchengines, such as Collecta, have already shown

intention of licensing their API as a revenue source.

Sponsoring discussion around live events - Bytracking hashtags on Twitter, we’re already seeingpeople using a live event’s real-time content as acontent channel. It will be even more customizedlike what we’ve already seen with The Huf fingtonPost and their real-time reporting around The WorldSeries. They created a content page that includedreal-time programming from YouTube, Twitterhashtag searches, and tweets from a select groupof sports reporters. Just like event producers getsponsors to put up signage or a booth at a sporting

event or conference. The same type of sponsorshipopportunities are also possible for real-timeprogramming.


Curated live searches - Real-time search can helpimprove the discovery of current discussions, butlike with any programming there will be a need tohave that content produced by an actual humanthat has an editorial eye. For example, a 24-hourwebcam pointed at the street streaming video tothe web will get some traf fic, but once a producerlooks at that video and edits it down to a shorterprogram it has more value, attracting a larger

audience. The same is true with real-time searchresults. Turning on a fire hose to spit out real-timesearch results can be interesting. Add a humaneditorial production layer on those search resultscan create exponentially more value. We’ve seenexamples of this already with blog posts touting

“best tweets” on a given subject or event (e.g. “The 10 best tweets (so far) from Microsoft’s MVP Summit” ).

This is another step beyond just sponsoringa discussion around a live event. If managedcorrectly, a human-powered curated live search

becomes another programming channel againstwhich to sell advertising.

(cont’d)

Emerging business models for real-time search


http://blogs.zdnet.com/microsoft/?p=2209






Page 17


Syndicated content feeds - Once a real-timesearch is created, it can then be syndicated tomultiple locations. Producers can add value layeringreal-time searches with human-powered curationand a custom interactive display of results. Brandsand sites will pay for this content to bring traf fic totheir site, especially during live events.


Customer service and CRM - Currently, more than 100 companies offer some type of socialmedia monitoring solution. All have some typeof real-time search component. Some are free,while some require a monthly subscription thatbusinesses happily pay just to know what peopleare saying about them. Most just provide alertsand summarized reports. Others are verticallyintegrated with real-time communications tools

(e.g. TweetDeck, PeopleBrowsr, Seesmic) allowingusers to respond back. Some tools (e.g. CoTweet,HootSuite) allow companies to manage responsesacross a team of people. When marketers seea real-time discussion, they can make a timelyresponse. To improve the customer serviceexperience, expect to see a consolidation of thesethree types of applications: real-time searchengines, social media monitoring, real-timecommunications tools.

Emerging business models for real-time search (cont’d)


http://wiki.kenburbary.com/






http://tweetdeck.com/

http://peoplebrowsr.com/


http://seesmic.com/

http://cotweet.com/

http://hootsuite.com/

http://cotweet.com/

http://hootsuite.com/

http://seesmic.com/


http://tweetdeck.com/




Page 18

What should we expect from real-time search

1. Real-time search engines that are only indexingTwitter (not including Twitter search) will haveto expand their offerings to scan other real-timecontent and to include implicit behavior.

2. Real-time search will soon be seen as a validprogramming channel that will include human-powered search curators that will produce morevaluable real-time productions.

3. Consumers and marketers are slowly realizingthe value of using real-time search as acustomer relations management tool. Whypick up the phone and complain to Comcast,when you can just send out a public tweetcomplaining? They’ll call you. It’s alreadyhappening. To facilitate the process, we’llsee a consolidation of the tools being usedfor monitoring and interaction (e.g. Radian6,Tweetdeck, CoTweet) with the real-time web.

4. Search advertising will be a big boon, asmarketers will be able to send multiple targetedmessages in a given day.

As for all the parties involved, here are somerecommendations:

Investors - Avoid investing in real-time searchengines that are completely based on Twitter’s API.You don’t want to invest in a company that’s fullybased on a startup. Insttant is a search engine builton Twitter’s API. CEO Joe Langevin admits thathis biggest fear is Twitter releasing its own killer

application. Act quickly, sources fear that real-timesearch could become a commodity in three to fiveyears.


Startups - Just being able to digest informationin a cool way probably won’t be enough. Only onecompany that’s doing it the best, possibly Scoopler,will succeed. The real winners will be presentinga new way to index information (e.g. OneRiot),will be offering another experience like a real-timeprogramming channel (e.g. Collecta), or will offeran application that takes advantage of real-timeinformation in a unique and valuable way.

Incumbents - Through their partnerships withTwitter and Facebook, Microsoft and Google havemade a good first step that will save them frominitially losing the 15-40 percent of people seekingreal-time results. But they’ll need to expand theirconcept of search to include discovery and contentprogramming.

Gatekeepers - Facebook will reveal a real-timesearch tool that will invite more content and activityinto its walled garden. Since they have a criticalmass of activity, they won’t need to make any other

partnerships or bring in other real-time activity.Twitter may be fine making a deal with Microsoftand Google as people will be satisfied with thoseresults and not bother to jump to other “real-timeonly” search engines.

This is just the beginning. Search is a learnedbehavior. Ultimately, how people will interact withreal-time search will be a symbiotic educationfor the real-time search engine creators and theconsumers. User behavior will evolve and will craftthe presentation, application, and redefinition of real-time search. We expect that with the expanded

definition of real-time search, we may startreferring to it as something else such as real-timeprogramming.




Page 19

i “Why 1% of search market share is worth over $1 Billion” byDon Dodge (2007).

ii Combination of content from GigaTweet and AddToAny’s report as reported by Mashable on distribution of real-time content forits web sharing tool.

iii Analysis extrapolated by Tobias Peggs, CEO of OneRiotand Gerry Campbell, CEO of Collecta from three researchreports on the study of intent of when people use a searchengine: “A Taxonomy of Web Search” by Andrei Broder, 2002,

“Understanding User Goals in Web Search” by Rose and Levinson(2004), and “Determining the informational, navigational, and transactional intent of Web queries” by Jim Jansen at Penn State(2007).

iv “Sharing on Facebook Now More Popular Than Sharing on Email” by Adam Ostrow of Mashable, sourcing AddToAny (July,2009)

v “New Twitter Research: Men Follow Men and Nobody Tweets” byBill Heil and Mikolaj Piskorski (June, 2009)

vi “Do You Know Who’s on Twitter?” by eMarketer (August, 2009)


Endnotes


http://dondodge.typepad.com/the_next_big_thing/2007/05/why_1_of_search.html


http://mashable.com/2009/07/20/facebook-sharing-data/







http://www.sigir.org/forum/F2000/broder.pdf

http://portal.acm.org/citation.cfm?id=988675

http://ist.psu.edu/faculty_pages/jjansen/academic/pubs/jansen_user_intent.pdf









http://blogs.harvardbusiness.org/cs/2009/06/new_twitter_research_men_follo.html

http://www.emarketer.com/Article.aspx?R=1007250


http://portal.acm.org/citation.cfm?id=988675

http://www.sigir.org/forum/F2000/broder.pdf

http://www.emarketer.com/Article.aspx?R=1007250

http://blogs.harvardbusiness.org/cs/2009/06/new_twitter_research_men_follo.html



http://popacular.com/gigatweet




David Spark is the founder of Spark Media Solutions, a firm that helps companies buildindustry voice through storytelling and socialmedia. For more than 14 years, Spark has workedas a journalist and producer reporting on thetech industry in print, radio, TV, and online. Hisarticles and advice have appeared in more than30 media outlets including eWEEK, Wired News,PC Computing, PC World, Smart Computing,

and TechTV. He’s worked as an analyst for TheGuidewire Group and Gerson Lehrman Group.

In addition, Spark squandered more than a dozenyears working as a stand-up comedian and comedywriter for Second City comedy troupe in Chicago. Today, Spark is a regular contributor to Mashable,Socialmedia.biz, Green 960, ABC Radio in NYC,John C. Dvorak’s “Cranky Geeks,” and KQED’s “ThisWeek in Northern California.” For more on Spark,follow his blog the “Spark Minute.”


Contact: [email protected]

Author: David Spark

Spark Media Solutions helps businesses becomemedia networks as an arm of its marketing, sales,and public relations effort. We are a custompublishing organization that provides editorialguidance and production across all forms of media.Our goal is to turn our clients into the numberone voice in their respective industries throughstorytelling and social media. The content and theconversation that surrounds our efforts is designed

to position our clients as thought leaders in theirrespective industry.

Spark Media Solutions

© 2009, Spark Media Solutions, LLC.

http://www.sparkmediasolutions.com/






http://sparkminute.com/


http://sparkminute.com/

Documents

Real-time Search and Discovery of the Social Web