Text Analytics Past, Present & Future: An Industry View

  • Published on
    21-Apr-2017

  • View
    2.680

  • Download
    1

Embed Size (px)

Transcript

Text Analytics Applied

Text Analytics Past, Present & Future: An Industry View Seth GrimesAlta Plana Corporation@sethgrimesJune 5, 2014

Copyright 2002 Alta Plana Corporation1

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Analytics is the systematic application of algorithmic methods that derive and deliver information, typically expressed quantitatively, whether in the form of indicators, tables, visualizations, or models.

Systematic means formal & repeatable.Algorithmic contrasts with heuristic.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Text analytics past:

Pioneers

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Document input and processingKnowledge handling is key

Desk Set (1957): Computer engineer Richard Sumner (Spencer Tracy) and television network librarian Bunny Watson (Katherine Hepburn) and the "electronic brain" EMERAC.Hans Peter Luhn A Business Intelligence SystemIBM Journal, October 1958

Copyright 2002 Alta Plana Corporation1

Statistical information derived from word frequency and distribution is used by the machine to compute a relative measure of significance, first for individual words and then for sentences. Sentences scoring highest in significance are extracted and printed out to become the auto-abstract.H.P. Luhn, The Automatic Creation of Literature Abstracts, IBM Journal, 1958.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Pipelines and patternsIBMs MedTAKMI, 1997-

http://www.research.ibm.com/trl/projects/textmining/index_e.htm

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Exhaustive extractionAn (old) Attensity example NLP to identify roles and relationships, for a law-enforcement application .

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Language engineeringGATE: General Architecture for Text Engineering.

http://gate.ac.uk/

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Text analytics present:

Business, technology, applications, and solutions

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Organizations embracing text analytics all report having an epiphany moment when they suddenly knew more than before. -- Philip Russom, the Data Warehousing Institute, 2007http://tdwi.org/articles/2007/05/09-what-works/bi-search-and-text-analytics.aspx

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Linguistics, statistics, and semanticsText analytics (typically) involves linguistic modelling, statistical characterization, learned patterns, and semantic understanding of text-derived features Named entities: people, companies, places, etc.Pattern-based features: e-mail addresses, phone numbers, etc.Concepts: abstractions of entities.Facts and relationships.Events.Concrete and abstract attributes (e.g., expensive & comfortable) including measure-value pairs.Subjectivity in the forms of opinions, sentiments, and emotions: attitudinal data. applied to business ends.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

SourcesIts a truism that 80% of enterprise-relevant information originates in unstructured form:E-mail and messages.Web pages, online news & blogs, forum postings, and other social media.Contact-center notes and transcripts.Surveys, feedback forms, warranty claims.Scientific literature, books, legal documents....Non-text unstructured content?ImagesAudio including speechVideoValue derives from patterns.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

ValueWhat do we do with text, whether online, on-social, or in the enterprise?Post/Publish, Manage, and Archive.Index and Search.Categorize and Classify according to metadata & contents.Extract information and Analyze.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Semantics, analytics, and IRText analytics generates semantics to bridge search, BI, and applications, enabling next-generation information systems.

SearchBI/Big DataApplica-tionsSearch based applications (search + text + apps)Information access (search + analytics)Synthesis (text + BI)/(big data)

Text analytics (inner circle)Semantic search (search + text)NextGen CRM, EFM, MR, marketing, apps

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Content, composites, connections 1

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Content, Composites, Connections, 2Content, composites, connections 2

Text Analytics: An Industry ViewJADT June 5, 2014#ApplicationsText analytics has applications in:Intelligence & law enforcement.Life sciences & clinical medicine.Media & publishing including social-media analysis and contextual advertizing.Competitive intelligence.Voice of the Customer: CRM, product management & marketing.Public administration & policy.Legal, tax & regulatory (LTR) including compliance.Recruiting.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Opinion, sentiment & emotion

Text Analytics: An Industry ViewJADT June 5, 2014#Sentiment analysisA specialization, of relevance to:Brand/reputation management.Customer experience management (CEM).Competitive intelligence.Survey analysis (EFM = Enterprise Feedback Management).Market research.Product design/quality.Trend spotting.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Data exploration via dashboards and workbenches.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Text analytics present:

The market

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

http://altaplana.com/TA2014

Text Analytics: An Industry ViewJADT June 5, 2014#

Text Analytics: An Industry ViewJADT June 5, 2014#Voice of the CustomerText analytics is applied to improve customer service and boost satisfaction and loyalty.Analyze customer interactions and opinions E-mail, contact-center notes, survey responses.Forum & blog posting and other social media. to Address customer product & service issues.Improve quality.Manage brand & reputation.Assessment of qualitative information from text helps users Gain feedback on interactions.Assess customer value.Understand root causes.Mine data for measures such as churn likelihood.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

The commercial scene

Text Analytics: An Industry ViewJADT June 5, 2014#Online commerceText analytics is applied for marketing, search optimization, competitive intelligence.Analyze social media and enterprise feedback to understand the Voice of the Market: OpportunitiesThreatsTrendsCategorize product and service offerings for on-site search and faceted navigation and to enrich content delivery.Annotate pages to enhance Web-search findability, ranking.Scrape competitor sites for offers and pricing.Analyze social and news media for competitive information.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

E-Discovery and complianceText analytics is applied for compliance, fraud and risk, and e-discovery.Regulatory mandates and corporate practices dictate Monitoring corporate communicationsManaging electronic stored information for production in event of litigationSources include e-mail (!!), news, social mediaRisk avoidance and fraud detection are key to effective decision makingText analytics mines critical data from unstructured sourcesIntegrated text-transactional analytics provides rich insights

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Text Analytics: An Industry ViewJADT June 5, 2014#

Text Analytics: An Industry ViewJADT June 5, 2014#

Text Analytics: An Industry ViewJADT June 5, 2014#The share rise in users who selected Arabiccoincided with much of the civil unrest in Middle Eastern countries.http://bits.blogs.nytimes.com/2014/03/09/the-languages-of-twitter-users/

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Non-English language support?

Text Analytics: An Industry ViewJADT June 5, 2014#Software & platform optionsText-analytics options may be grouped in general classes.Installed text-analysis application, whether desktop or server or deployed in-database.Data mining workbench.Hosted.Programming tool.As-a-service, via an application programming interface (API).Code library or component of a business/vertical application, for instance for CRM, e-discovery, search.Text analytics is frequently embedded in search or other end-user applications.The slides that follow next will present leading options in each category except Hosted

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Text Analytics: An Industry ViewJADT June 5, 2014#User decision criteriaPrimary considerations include Adaptation or specialization: To a business or cultural domain, language, information type (e.g., text, speech, images) & source (e.g., Twitter, e-mail, online news).By-user customization possibilities: For instance, via custom taxonomies, rules, lexicons.Sentiment resolution: Aggregate, message, or feature level. (What features? Topics, coreferenced entities?)What sentiment? Valence & what else? Emotion? Intent?Outputs: E.g., annotated text, models, indicators, dashboards, exploratory data interfaces.Usage mode: As-a-service (API), installed, or hosted/cloud.Capacity: Volume, performance, throughput, latency.Cost.

Text Analytics: An Industry ViewJADT June 5, 2014#A few French companies

Text Analytics: An Industry ViewJADT June 5, 2014#Academic spin-offs

People Pattern

Text Analytics: An Industry ViewJADT June 5, 2014#Text analytics future:

Synthesis and sensemaking.

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

New York Times,September 8, 1957

Text Analytics: An Industry ViewJADT June 5, 2014#

Emotion in text

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Emotion and outcomes

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Audio including speech.Images.Video.

http://www.geekosystem.com/facebook-face-recognition/

http://www.sciencedirect.com/science/article/pii/S0167639312000118

http://flylib.com/books/en/2.495.1.54/1/Beyond Text

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

The world of big dataMachine data (e.g., logs, sensor outputs, clickstreams).Actions, interactions, and transactions: geolocation and time.Profiles: individual, demographic & behavioral.Text, audio, images, and video.

Facts and feelings.

Text Analytics: An Industry ViewJADT June 5, 2014#

(Accessible) data everywhere

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

http://www.geeklawblog.com/2011/12/lexis-advance-platform-launch-two.html

A big data analytics architecture (example)

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

http://searchuserinterfaces.com/It is convenient to divide the entire information access process into two main components: information retrieval through searching and browsing, and analysis and synthesis of results. This broader process is often referred to in the literature as sensemaking. Sensemaking refers to an iterative process of formulating a conceptual representation from of a large volume of information. Marti Hearst, 2009

Sensemaking

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

http://www.businessweek.com/magazine/content/04_19/b3882029_mz072.htmEn route

Text Analytics: An Industry ViewJADT June 5, 2014#

Copyright 2002 Alta Plana Corporation1

Text Analytics Past, Present & Future: An Industry View Seth GrimesAlta Plana Corporation@sethgrimesJune 5, 2014

Copyright 2002 Alta Plana Corporation1

Recommended

View more >