Hadoop @ eBay: Past, Present and Future

  • Published on
    24-Feb-2016

  • View
    81

  • Download
    0

Embed Size (px)

DESCRIPTION

Hadoop @ eBay: Past, Present and Future. Ryan Hennig Hadoop Platform Team. ABOUT ME. RYAN HENNIG. Born and raised in Seattle, WA Studied Computer Science at University of Washington in Seattle Worked on Microsoft SQL Server 2006 2012 Shipped SQL Server 2008, 2008 R2, 2012 - PowerPoint PPT Presentation

Transcript

HEADLINE IN ARIAL REGULAR 39PT, ALL UPPERCASE

Hadoop @ eBay: Past, Present and FutureRyan HennigHadoop Platform Team

ABOUT MERYAN HENNIGBorn and raised in Seattle, WA

Studied Computer Science at University of Washington in Seattle

Worked on Microsoft SQL Server 2006 2012Shipped SQL Server 2008, 2008 R2, 2012

Joined eBay Hadoop team in early 2012 - Based in Bellevue, suburb of SeattleCOMPUTE AND DATA INFRASTRUCTURE3

AGENDAPast: Growth of Hadoop at eBayPresent: Hadoop Use Cases, Operations ToolsFuture: Hadoop 2.0

HADOOP AT EBAY: PASTGrowth of Hadoop at eBay

Adventures in Forking

Partnership with HortonworksHADOOP EVOLUTION @ eBayHADOOP AT EBAY: PAST62007Single digit nodes2010Shared cluster100s nodes1000s + corePBCDH22011Shared clusters1000s node10,000+ core10s PBWilma (0.20)

2012Shared clusters1000s node10,000+ core10s PBArgon (0.22)

2013Shared clusters 4k+ node40,000+ core50s PBHDP 1.x

2009Search10s- nodes

ADVENTURES IN FORKING2007-2010: eBay runs shared clusters on Cloudera Distribution of Hadoop

2010-2012: eBay runs shared clusters on custom Hadoop versions2010: Wilma (based on 0.20)2011: Argon (based on 0.22)2012: Custom branch abandoned

Lessons LearnedForking a fast-changing open source project is difficult and riskyBalancing Development and operations needsDevelopment team sizeFacebook had 100eBay had 15Coordination with open source community = lots of overheadDivergence from open source: Push changes early and often

HADOOP AT EBAY: PAST7HADOOP AT EBAY: PAST8

EBAY AND HORTONWORKS2012: eBay enters partnership with HortonWorksGoalsFocus on eBay-specific development internallyLeverage HortonWorks expertise for general Hadoop DevelopmentAvoid source code divergence by making open source contribution a priorityBenefits to HortonWorksCredibility enhanced by having a well-known customerAbility to test at large scale

HADOOP AT EBAY: PAST9HADOOP AT EBAY:PRESENTShared and Dedicated Clusters

Job Distribution

Use Case Examples

eBay Data Platform Overview

SHARED AND DEDICATED CLUSTERSShared clusters10s of PB and 10s of thousands of slots per clusterUsed primarily for analytics of user behavior and inventoryMix of production and ad-hoc jobsMix of MR, Hive, PIG, Cascading etc.Hadoop and HBase security enabled

Dedicated clustersVery specific use cases like Index BuildingTight SLAs for jobs (in order of minutes)Immediate revenue impactUsually smaller than our shared clusters, but still big (100s of nodes)

HADOOP AT EBAY: PRESENT11JOB DISTRIBUTION BY TYPEHADOOP AT EBAY: PRESENT12

USE CASE EXAMPLESCassini, eBays new search engine:Use MR to build full and incremental near-real-time indexesRaw Data is stored in HBase for efficient updates and random readStrong SLAs: < 10 minutesRun on dedicated clusters

Related and similar Items recommendations:Use transactional data, click stream data, search index, etc.Production MR jobs on a shared cluster

Analytics dashboard:Run Mobius MR jobs to join click stream data and transactional data Store summary data in HBaseWeb application to query HBaseHADOOP AT EBAY: PRESENT13HADOOP OPERATIONSLDAP Integration- All users stored in Active Directory, accessed via LDAP- Access to MapReduce Queues granted via MapReduce queues- Batch users: shared by a group of users

Security- Kerberos as implemented by Microsoft Active Directory- One domain for users, another for service/server principals - Batch users authenticated via keytabs, not passwords

Misc- 10s of slave nodes are broken at any given time- Often need to add several racks of machines at a timeHADOOP AT EBAY: PRESENT14HADOOP OPERATIONSTeam has Development and Operations Responsibilities2 Huge shared clusters1800+ users, exponential growthAbout 10 Hadoop developersRecently: operations work moved to dedicated team

Developed several tools to manage operationsHadoop Management Console: user-facing web appldap-admin: swiss-army knife style tool for hadoop adminsPuppet: for adding machines to the clusters, many racks at a timeDecom/Recom scripts: automatic detection, repair, decommission, and recommission of slave nodes

HADOOP AT EBAY: PRESENT15HADOOP MANAGEMENT CONSOLECustom Web application built on Ruby on RailsSelf-service tools are continually added to reduce support loadUser ManagementAccess RequestsGroup MembershipBatch User ManagementNew RequestsSudoer managementDataset ManagementExplore DatasetsRequest New dataset transfer between Teradata and HadoopMetadata toolsEach dataset is stored in custom XML formatCode Generation: Hive Tables, Java POJOs

HADOOP AT EBAY: PRESENT16HADOOP AT EBAY: PRESENT17

HADOOP AT EBAY: PRESENT18

HADOOP AT EBAY: PRESENT19

HADOOP AT EBAY: PRESENT20

HADOOP AT EBAY: PRESENT21

HADOOP AT EBAY: PRESENT22

HADOOP AT EBAY: PRESENT23

ldap-adminCommand-line tool written in RubySwiss-army knife tool, features added on demand for support issuesOften used features:Add a user to a groupView key details for LDAP users and groupsList all users, batch users, hadoop groupsReset batch user passwords and keytabsShow/add/remove sudoers for a batch accountRun user diagnostics: check permissions, keytabs, etc

HADOOP AT EBAY: PRESENT24HADOOP AT EBAY:FUTUREHDFS Federation

YARN

New Scenarios

Storage and Operational Efficiency

HDFS HA and FederationHDFS High-Availability for ReliabilityNameNode in Hadoop 1.0 is a Single Point of FailureAutomated failover to hot standby Depends on ZooKeeper

HDFS Federation for Scalability and IsolationHadoop 1.0: Single NameNode serviceSecondary NameNode is not for failoverStorage scales horizontally, but Namespace scales verticallyNo isolation for different tenants or applications

Hadoop 2.0: HDFS FederationPartition the HDFS Namespace Many independent NameNodesAllows direct access to Block Storage w/o going through HDFS interfaceHADOOP AT EBAY: FUTURE26HDFS HAHADOOP AT EBAY: FUTURE27

HDFS HAHADOOP AT EBAY: FUTURE28

HDFS HAHADOOP AT EBAY: FUTURE29

HDFS FederationHADOOP AT EBAY: FUTURE30

Horizontal Scalability of HDFS Namespace

Multiple independent NameNodes serving a subtree of the NameSpace

Example: NN1 provides /users, NN2 provides /reportsYARNHADOOP AT EBAY: FUTURE31Hadoop 1.0: MapReduceJobTracker and TaskTracker servicesHandles Resource Management, Job Execution

Hadoop 2.0: YARNRefactoring Responsiblities of JobTracker and TaskTracker into more general platformGlobal ResourceManagerCluster-wide resource managementsPer-application ApplicationMasterApplication-specific job controlYARN

HADOOP AT EBAY: FUTURE32YARNHADOOP AT EBAY: FUTURE33

YARNHADOOP AT EBAY: FUTURE34

YARNHADOOP AT EBAY: FUTURE35

New ScenariosIterative QueryStinger (Hive), Impala, etcRapid Data exploration and analysisGraph DatabasesTitanDB, Giraph Billions of vertices and edgesComplex Graph TraversalsApplications: PayPal fraud detection, Social Graph AnalysisReal-Time ProcessingStorm (Twitter), Apache S4Reinforcement Learning, Monitoring

HADOOP AT EBAY: FUTURE36Efficiency and ReliabilityStorage EfficiencyHDFS introduces a 3x storage cost for its replicasHDFS-RAID: more reliability for 1.5x storage costReed-SolomonLocally Repairable Codes (Project Xorbas)Tradeoff: the cost of repairing lost data is much higherOperational EfficiencyMore automationMore self-service toolsBetter Monitoring

HADOOP AT EBAY: FUTURE37Open SourceHMC MetadataLong term goal: standardize on open source technologies (HCatalog)Short term: explore what should be open sourcedHadoop Management ConsoleHadoop Access Request AutomationBatch user creation and managementMetadata managementCode generation of dataset to Hive tables and Java POJOsldap_admin toolsVery useful but tightly coupled to eBays LDAP configurationWilling to open source if there is interest

HADOOP AT EBAY: FUTURE38THANK YOU

Questions?