NYC HUG - Application Architectures with Apache Hadoop

1

Headline Goes Here Speaker Name or Subhead Goes Here

DO NOT USE PUBLICLY PRIOR TO 10/23/12

ApplicaAon Architectures with Apache Hadoop Mark Grover | @mark_grover NYC HUG slideshare.com/markgrover October 14th, 2014

©2014 Cloudera, Inc. All Rights Reserved.

About Me •  CommiPer on Apache Bigtop, commiPer and PPMC member on Apache Sentry (incubaAng).

•  Contributor to Hadoop, Hive, Spark, Sqoop, Flume. •  SoWware developer at Cloudera • @mark_grover

2 ©2014 Cloudera, Inc. All Rights Reserved.

Co-‐authoring O’Reilly book

• @hadooparchbook •  hadooparchitecturebook.com •  Strata Hadoop World Tutorial

•  at 9 AM tomorrow

©2014 Cloudera, Inc. All Rights Reserved. 3

4

Click Stream Analysis

Case Study


AnalyAcs


AnalyAcs


AnalyAcs


AnalyAcs


AnalyAcs


AnalyAcs


AnalyAcs


Web Logs – Combined Log Format


244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36” 244.157.45.12 - - [17/Oct/2014:21:59:59 ] "GET /Store/cart.jsp?productID=1023 HTTP/1.0" 200 3757 "http://www.casualcyclist.com" "Mozilla/5.0 (Linux; U; Android 2.3.5; en-us; HTC Vision Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1”

12

Clickstream AnalyAcs


244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36”

13

Similar use-‐cases

•  Sensors – heart, agriculture, etc. •  Casinos – session of a person at a table


Challenges of Hadoop ImplementaAon


Challenges of Hadoop ImplementaAon


Other challenges -‐ Architectural ConsideraAons

•  Storage managers? •  HDFS? HBase?

•  Data storage and modeling: •  File formats? Compression? Schema design?

•  Data movement •  How do we actually get the data into Hadoop? How do we get it out?

•  Metadata •  How do we manage data about the data?

•  Processing •  How can we transform it? How do we query it?

•  OrchestraAon •  How do we manage the workflow for all of this?


18

Since that’s all what the Ame allows todayJ

2. Processing


Processing

• De-‐duplicaAon •  Filtering •  SessionizaAon


DeduplicaAon – remove duplicate records


244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36” 244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36”

Filtering – filter out invalid records


244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36” 244.157.45.12 - - [17/Oct/2014:21:59:59 ] "GET /Store/cart.jsp?productID=1023 HTTP/1.0" 200 3757 "http://www.casualcyclist.com" "Mozilla/5.0 (Linux; U…

SessionizaAon


Website visit

Visitor 1 Session 1

Visitor 1 Session 2

Visitor 2 Session 1

> 30 minutes

Why sessionize?

Helps answers quesAons like: • What is my website’s bounce rate?

•  i.e. how many % of visitors don’t go past the landing page? • Which markeAng channels (e.g. organic search, display ad, etc.) are leading to most sessions?

• Which ones of those lead to most conversions (e.g. people buying things, signing up, etc.)

• Do aPribuAon analysis – which channels are responsible for most conversions?


SessionizaAon


244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36” 165 244.157.45.12 - - [17/Oct/2014:21:59:59 ] "GET /Store/cart.jsp?productID=1023 HTTP/1.0" 200 3757 "http://www.casualcyclist.com" "Mozilla/5.0 (Linux; U; Android 2.3.5; en-us; HTC Vision Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1” 166

How to Sessionize?

1.  Given a list of clicks, determine which clicks came from the same user

2.  Given a parAcular user's clicks, determine if a given click is a part of a new session or a conAnuaAon of the previous session


#1 – Which clicks are from same user?

• We can use: •  IP address (244.157.45.12) •  Cookies (A9A3BECE0563982D) •  IP address (244.157.45.12)and user agent string ((KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36")



• We can use: •  IP address (244.157.45.12) •  Cookies (A9A3BECE0563982D) •  IP address (244.157.45.12)and user agent string ((KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36")





#2 – Which clicks part of the same session?



> 30 mins apart = different sessions

30

github.com/hadooparchitecturebook/clickstream-‐tutorial

SessionizaAon in MapReduce


SessionizaAon in MapReduce


Map

Reduce

Reduce

Log line IP1, log lines

IP1, log line

s

Log line, session ID

Map

Map

Log line

Log line IP2, log lines

IP2, log lines Log line, session ID

Mapper for SessionizaAon


public static class SessionizeMapper extends Mapper<Object, Text, IpTimestampKey, Text> { private Matcher logRecordMatcher; public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { logRecordMatcher = logRecordPattern.matcher(value.toString()); // We only emit something out if the record matches with our regex. Otherwise, we assume the record is busted and simply ignore it if (logRecordMatcher.matches()) { String ip = logRecordMatcher.group(1); DateTime timestamp = DateTime.parse(logRecordMatcher.group(2), TIMESTAMP_FORMATTER); Long unixTimestamp = timestamp.getMillis(); IpTimestampKey outputKey = new IpTimestampKey(ip, unixTimestamp); context.write(outputKey, value); } } }

Reducer for SessionizaAon


public static class SessionizeReducer extends Reducer<IpTimestampKey, Text, IpTimestampKey, Text> {

private Text result = new Text();

public void reduce(IpTimestampKey key, Iterable<Text> values, Context context

) throws IOException, InterruptedException { // The sessionId generated here is per day, per IP. So, any queries // that will be done as if this session ID were global, would require // a combination of the day in question and IP as well. String sessionId = null; Long lastTimeStamp = null; for (Text value : values) { String logRecord = value.toString();

Reducer for SessionizaAon


// If this is the first record for this user or it's been more than the timeout since // the last click from this user, let's increment the session ID. if (lastTimeStamp == null || (key.getUnixTimestamp() -‐ lastTimeStamp > SESSION_TIMEOUT_IN_MS)) { sessionId = key.getIp() + "+" + key.getUnixTimestamp(); } lastTimeStamp = key.getUnixTimestamp(); result.set(logRecord + " " + sessionId); // Since we only care about printing out the entire record in the result, with session ID appended // at the end, we just emit out "null" for the key context.write(null, result); } } }

Secondary sorAng – by Amestamp

• Need records to reducer to be grouped by IP address and sorted by Amestamp – a concept called secondary sor/ng

•  Instead of using just IP address as map output key and reduce input key

• We use a composite key (IP, Amestamp) as map output key and reduce input key


Secondary sorAng – vocabulary

•  Composite key – IP address, Amestamp • Natural key – IP address •  Secondary sort key -‐ Amestamp


Secondary sorAng

•  Custom Grouping Comparator – on Natural Key (IP) •  Custom Sort Comparator – on Composite Key (IP, address) •  Custom ParAAoner – on Natural Key (IP) job.setGroupingComparatorClass(NaturalKeyComparator.class); job.setSortComparatorClass(CompositeKeyComparator.class); job.setPartitionerClass(NaturalKeyPartitioner.class);


38

Final Architecture



Hadoop Cluster

BI/VisualizaAon tool (e.g.

microstrategy)

BI Analysts

Spark For machine learning and graph processing

R/Python StaAsAcal Analysis

Custom Apps

3. Accessing

2. Processing

4. OrchestraAon via Oozie 1. IngesAon

OperaAonal Data Store

CRM System Via Sqoop

Web servers

Website users

Final Architecture – High Level Overview

40

Data Sources IngesAon

Data Storage/Processing

Data ReporAng/Analysis



41





Final Architecture – IngesAon

42

Web App Avro Agent Web App Avro Agent




Flume Agent

Flume Agent

Flume Agent

Flume Agent

Fan-‐in PaPern

MulA Agents for Failover and rolling restarts

HDFS



43





Final Architecture – Storage and Processing

44

/etl/weblogs/20140331/ /etl/weblogs/20140401/ …

Data Processing /data/markeAng/clickstream/bouncerate/ /data/markeAng/clickstream/aPribuAon/ …



45





Final Architecture – Data Access

46

Hive/Impala

BI/AnalyAcs Tools

DWH Sqoop

Local Disk

R, etc.

DB import tool

JDBC/ODBC


Contact info • Mark Grover

• @mark_grover •  www.linkedin.com/in/grovermark

•  Slides at slideshare.net/markgrover



Technology

NYC HUG - Application Architectures with Apache Hadoop