Embed Size (px)
DESCRIPTION
With the torrent of data available to us on the Internet, it's been increasingly difficult to separate the signal from the noise. We set out on a journey with a simple directive: Figure out a way to discover emerging technology trends. Through a series of experiments, trials, and pivots, we found our answer in the power of graph databases. We essentially built our "Emerging Tech Radar" on emerging technologies with graph databases being central to our discovery platform. Using a mix of NoSQL databases and open source libraries we built a scalable information digestion platform which touches upon multiple topics such as NLP, named entity extraction, data cleansing, cypher queries, multiple visualizations, and polymorphic persistence.
Citation preview
Discovering Emerging Technology Through Graph Analysis
GraphConnect | Chicago June 2013
About Me
[email protected] || [email protected]
@henry74
henry74
Founder / Director of PwC's Emerging Tech Lab
What is the Emerging Tech Lab?
We build stuff to help people get smart about applying technology to solve problems
● Founded 3 years ago to identify and experiment with new technologies relevant to but not widely adopted by the Enterprise
● Focuses on rapid prototyping & MVP build-outs for both tactical internal projects and more creative, exploratory ideas
● Permanent core team, but operates a rotational program for staff to provide them an opportunity for hands-on technical experience, learning agile & lean principles, and exposure to a startup-like environment
The Challenge
It usually starts with an idea…
“Build a platform to help discover emerging technologies.”
…followed by some pretty mock-ups…
…to raise expectations.
Envisioning success
● What are some emerging technologies?
● How are they being used to solve real problems?
● Who is talking about them?● Who are the players?● Are there related technologies?
● Get up to speed quickly ● Discover related topics ● Understand what is trending● Find interesting applications● See what's possible
What makes technology “emerging”?
● Cannot already be mainstream technology
● Needs to be more than a single event to be an emerging trend
● Must be growing in popularity, but not yet popular
● "Technology" could be a thing (e.g. nanotubes), but also an
aggregation or application of technologies (e.g. cloud
computing, quantified self)
The Journey
Initial design
Data Feeds (RSS)
Pull & Store Raw
Data
MongoDB
Analyze VisualizeSource
?Postgres
Breaking ground
● Natural Language Processing
● Named Entity Recognition
● ???
● ???
● ???
● ???
● ???
Extract Text
Understand Text
Discover Insights
A bit more clarity
Data Feeds (RSS)
Pull & Store Raw
Data
MongoDB
Analyze VisualizeSource
?
3rd Party APIs
Tag & Update
Postgres
Digging a little deeper
● Natural Language Processing
● Named Entity Recognition
● Collocation?
● K-means clustering?
● Information Ontology?
● ???
● ???
Extract Text
Understand Text
Discover Insights
The Eureka moment...
…took a bit longer than it should have
Graphs are everywhere
Final design
Data Feeds (RSS)
Pull & Store Raw
Data
MongoDB
Analyze VisualizeSource
3rd Party API
Tag & Update
Neo4j Postgres
Lesson #1 - Graph data modeling is iterative
What should be a node, relationship, or a property? Depends on:● What will you search on? ● How do you start your searches?● How much data do you expect to have? What data?
Expect to change your graph based on:● Experimentation● Query syntax available to extract and aggregate graph data ● Query performance
TIP: Plan to reload your graph many times - save the raw data, start small,
use batch loading until you get it right
…but more flexible than traditional data modeling
Modeling the data
DOC
P
P
C
K
K
C
T
C
DOC
P
P
C
K
K
O
T
Document are described by its entities, concepts, and keywords through relationships
This means:
● Document are related to other documents through shared entities, concepts, and keywords
● Concepts and entities are related to each other through shared documents
● Incoming relationships measures # of referring documents
Simple, yet powerful
TAGGED_AS
RELATES_TO
REFERS_TO
CONTAINS
REFERS_TO
Lesson #2 - Connections are important
Highly connected data creates richer graphs and increases potential for discovering greater insights
BUT unnecessary connections can create noise & extra work
Don't create artificial connections, but clean up data before importing when it makes sense (e.g. networking, networks, network)
Prevent duplication which can impact your insights based on aggregation (e.g. # of relationships) or certain patterns
Keeping it clean
Techniques Graph Benefits
Text extraction with readability scoring
● Better named entity extraction● Improve neighbor relevance● Minimize invalid nodes & relationships
Similarity Hashing ● Improve validity of relationships● Increase graph connectedness
Porter Stemming ● Improve graph connectedness
Lesson #3 - Understand Cypher
● Cypher experimentation opens up the possible● SQL users will be at home - tabular results, similar
syntax● Start without parameters, check with Neo4j shell,
move to parameterized queries for security & performance (caching)
● Don't forget Lucene syntax● Continues to evolve for the better - check new release
changes (http://docs.neo4j.org/refcard/1.9/)
● Let Cypher do the work
Useful Cypher Syntax
START with an indexMATCH defines your universeWHERE filters it downWITH combines multiple statementsHAS checks if a property existsAS lets you name your return valuesIN checks against an arrayCOLLECT aggregates into an arrayORDER just like SQLLIMIT for performance
Prototype highlights
● 4 people & 4 months (first version)● Data Stores - Neo4J, MongoDB, Redis, Postgres● Visuals - D3.js, Vivagraph.js, Twitter Bootstrap● Key Languages/Libraries - Ruby, Rails, Cypher,
Knockout.js, Amplify.js, HTML5, CSS3, jQuery, Neography gem, Resque gem
● 3rd Party - Alchemy, OpenCalais, RSS feeds, Wikipedia
● Concepts - natural language processing, named entity extraction, text cleansing & de-duplication (map/reduce), similarity hashing, large-scale information retrieval
● 1M+ nodes, 3M+ relationships, 6M+ properties after 6 months
Emerging Tech Radar Demo
Tag Cloud / Search
DOC C
K
K
C
DOC
C
K
K
DOC
DOC
DOC
DOC
● Index keywords and search across keywords (tip: use Lucene syntax)● Identify documents with strong relationships to keywords● Locate concepts with strongest relationships to relevant documents● Popularity based on number of incoming relationships
Emerging Index / Popularity / Doc List
DOC CDOC
(E) OC
DOC(NE)
DOC(E)
DOC(E)
DOC(NE)
DOC(E)
DOC(NE)
DOC(E)
Cloud computing (Concept) and Google (Org)
● Strong relationships with documents shared between concepts to filter and rank relevant content
● Ratio and strength of relationships to quantify emerging index● Popularity based on number of incoming relationships of each type of
document (emerging versus non-emerging)
Node Graph
DOC CK DOC OC
DOC
DOC
DOC
DOC DOC
DOC● Existing relationships with documents shared between concepts to
filter relevant neighbors● Strength of relationships based on # and weight for ranking relevance
(color)
C
The Takeaway
Final Thoughts
● Graphs makes it simple to generate complex insights - you don't need to be a data scientist
● Graphs are a natural fit for anything connected...which is most things (e.g. social media, internet of things, sensor data)
● Experimentation is the best way to learn the power of graphs
● Make graph databases a first class citizen in your technology toolkit - many things can be solved better with a graph
The best way to discover emerging technologies is to try them out
Thanks for Listening - Q & A
Special thanks to Max De Marzi for his neography gem (https://github.com/maxdemarzi/neography) and ongoing advice, suggestions, troubleshooting
