Double Your Hadoop Hardware Performance with SmartSense

1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Boost Apache Hadoop Hardware Performance 2X with SmartSense

Paul CoddingProduct Management Director

2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Hortonworks Connected Data Platforms and Solutions

Data Services

Hortonworks Solutions

Enterprise DataWarehouse Optimization

Cyber Security andThreat Management

Internet of Thingsand Streaming Analytics

Data CenterHortonworks Data Suite

HDFHDP

HortonworksConnection

CloudHortonworks Data CloudAWS HDInsight

Hortonworks ConnectionEnablement Subscription

SmartSense™

Premier Operational Support

Educational Services

Professional Services

Community Connection


Hortonworks Connection Ensures Success of Your Big Data Journey


5 Reasons Why You Need More Than Just Open Source Software

The open source community doesn’t ensure everything works together and is certified for the data center and cloud platforms you rely on. Hortonworks does.1The unprecedented pace of open source innovation is both a benefit and a challenge. Hortonworks can help; it’s what we do.2Your enterprise needs more than just support for the latest open source versions. Hortonworks supports and maintains the versions you rely on.3The community doesn’t ensure that consistent security, governance, and operations are built in. Hortonworks takes enterprise needs seriously.4The community is not responsible for your success with open source technologies and tools. Hortonworks success is built on your success. 5


Increase Performance

Prevent Issues

Accelerate Case Resolution

Understand Your Cluster


Issue: YARN @ capacity, struggling to add more use cases

Before SmartSense

Could only run 500 jobs concurrently

1100 jobs would be pending waiting for

resources at peak hours


After Applying only 3 SmartSense Recommendations

They can now run 1200 concurrent jobs

...with only 350 waiting jobs at peak hours


Before SmartSense









Before SmartSense




With SmartSense = 2X Throughput Improvement


Hardware ($$$)

Hadoop Performance

• Type of CPU & Core Count• Type & Amount of Memory• Type & Number of Disks


Hardware ($$$)

Operating System

Hadoop Performance


• Kernel Configuration• Disk Mount/Tuning• Network Configuration


Hardware ($$$)

Operating System

Hadoop Daemons

Hadoop Performance



• YARN/MR/Tez Memory Configuration• HDFS Configuration• ZooKeeper Configuration


Hardware ($$$)

Operating System

Hadoop Daemons

Hadoop Performance





What we do

A M B A R I

O P S S m a r t S e n s eS E R V E R

B U N D L EG AT E W AY

S m a r t S e n s eA n a l y ti c s

S m a r t S e n s eS E R V I C E

Collection Diagnostic Information Secure & Send Analyze & Recommend


Hardware ($$$)

Operating System

Hadoop Daemons

Hadoop Performance





YARN Memory Configuration

ContainersUnit of allocation for memory and compute

Scheduler Configuration Minimum Container Size Maximum Container Size

YARN NodeManager Configuration How much memory can be used by YARN on each cluster node

YARN Cluster1

2

3

4

5

6

7

64 GB




Scheduler Configuration Minimum Container Size: 5GB Maximum Container Size: 35GB


– 35GB

YARN Cluster1

2

3

4

5

6

7

64 GB



5

YARN ClusterApplication YARN Scheduler

I need 5 2GB containers

Min: 5GBMax: 35GB

201

2

3

4

5

6

7



Application YARN Scheduler


Min: 5GBMax: 35GB

5

5

5

5

5

YARN Cluster

5

201

2

3

4

5

6

7



Application YARN Scheduler


Min: 5GBMax: 35GB

Application is taking 25GB of resources when it only needs 10GB

5

5

5

5

5

YARN Cluster

5

201

2

3

4

5

6

7



Gatorade: $2.50Machine only takes CashEXACT CHANGE REQUIRED!



Gatorade: $2.50Machine only takes CashEXACT CHANGE REQUIRED!

Minimum Withdrawal: $20Maximum Withdrawal: $500




Scheduler Configuration Minimum Container Size: 2GB vs 5GB Maximum Container Size: 10GB vs 35GB


– 56GB vs 35GB

YARN Cluster1

2

3

4

5

6

7

64 GB






Before SmartSense




With SmartSense = 2X Throughput Improvement



Prevent Issues




Support Cases by Type

ConfigurationEnvironmentEducationNo ResponseProduct DefectUnreproducibleUse Case AdviceWorks as DesignedOther

SmartSense Today – Prevent Issues









30% of support cases are configuration issues—this is where SmartSense adds incredible value



SmartSense analyzes Bundles for configuration issues – recommendations are produced and made available for each cluster in the Hortonworks Support Portal

Recommendations prevent operational issues, and improve performance and overall cluster throughput.



SmartSense analyzes Bundles for configuration issues – recommendations are produced and made available for each cluster in the Hortonworks Support Portal

Recommendations prevent operational issues, and improve performance and overall cluster throughput.



Prevent Issues




SmartSense Today – Accelerate Case Resolution

SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support.

Significantly reduces the back-and-forth nature of troubleshooting issues.

A M B A R I

O P SH O R T O N W O R K S

S U P P O R T

S U P P O R TC A S E

S m a r t S e n s eS E R V E R






A M B A R I


S U P P O R T








A M B A R I


S U P P O R T








A M B A R I


S U P P O R T





SmartSense Data Capture Architecture

L A N D I N G Z O N E

S E R V E RG AT E W AY

A M B A R I

A G E N T A G E N T

A G E N TA G E N TA G E N T

A G E N T

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E






A M B A R I

A G E N T A G E N T


A G E N T

B U N D L E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

Agent to Server: TLS

Bundle: AES 256/RSA 1024

Landing Zone: SOC2 Certified






A M B A R I

A G E N T A G E N T


A G E N T

B U N D L E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E



Server to Gateway: TLS







A M B A R I

A G E N T A G E N T


A G E N T

B U N D L E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E



Server to Gateway: TLS


Gateway to Landing Zone: HTTPS (TLS 1.2) or SFTP (AES)




Prevent Issues




“Who’s creating all of these small files in HDFS!?”

“What are my top 10 most active users, and longest running jobs?”

“How much should I charge users for their cluster resource use?”

SmartSense Today – Understand Your Cluster


SmartSense Today – Understand Your ClusterOn-Premise

Chargeback Reporting




HDFS Dashboards




HDFS Dashboards

YARN Dashboards


Impact of Hortonworks SmartSense

Without SmartSense

With SmartSense

0200400600800

100012001400

Concurrent Jobs

B U N D L E

2X Throughput Improvement

Address 30% of Issues

Configuration Issues

Avoid 10% of Sev1 Issues

Production Down

Single-Bundle Case Resolution 25% of the Time

SmartSense Troubleshooting Bundle


Questions


Hortonworks Connected Data Platforms and Solutions

Data Services

Hortonworks Solutions

Enterprise DataWarehouse Optimization

Cyber Security andThreat Management

Internet of Thingsand Streaming Analytics

Data CenterHortonworks Data Suite

HDFHDP

HortonworksConnection

CloudHortonworks Data CloudAWS HDInsight

Hortonworks ConnectionEnablement Subscription

SmartSense™

Premier Operational Support

Educational Services

Professional Services

Community Connection

© DataWorks Summit and Hadoop Summit 2017. All Rights Reserved47

DataWorks Summit 2017

http://dataworkssummit.com

Technology

Double Your Hadoop Hardware Performance with SmartSense