28
1 © Cloudera, Inc. All rights reserved. クラウド上でのHadoop基盤 Cloudera Director 2.0 嶋内 翔、Cloudera

クラウド上でのHadoop基盤とCloudera Director 2.0 #rhcj2016

Embed Size (px)

Citation preview

  • 1 Cloudera, Inc. All rights reserved.

    HadoopCloudera Director 2.0 Cloudera

  • 2 Cloudera, Inc. All rights reserved.

    ()20114ClouderaClouderaemail: [email protected] twitter: @shiumachi

  • 5 Cloudera, Inc. All rights reserved.

    Cloudera Enterprise Hadoop Fast / Easy / Secure

    Cloudera: Fast : Easy : Secure :

    OPERATIONS DATA MANAGEMENT

    STRUCTURED UNSTRUCTURED

    PROCESS, ANALYZE, SERVE

    UNIFIED SERVICES

    RESOURCE MANAGEMENT SECURITY

    FILESYSTEM RELATIONAL NoSQL

    STORE

    INTEGRATE

    BATCH STREAM SQL SEARCH SDK

    Public Cloud Private Cloud Hybrid Environments

    Hybrid Deployment Flexibility

  • 6 Cloudera, Inc. All rights reserved.

    Hadoop

    Object Store

    STORE

    COMPUTE

  • 7 Cloudera, Inc. All rights reserved.

    Hadoop

    Hadoop

  • 8 Cloudera, Inc. All rights reserved.

    Cloudera: Hadoop

    CDHHadoop

    2009 2012 2013 2014 2015

    AWS

    Cloudera Enterprise MSP

    ClouderaAzure

    ClouderaGCP

    Hadoop

  • 9 Cloudera, Inc. All rights reserved.

    Hadoop

  • 11 Cloudera, Inc. All rights reserved.

    ETL/

    BI/

  • 15 Cloudera, Inc. All rights reserved.

    Easy:

    Launch Cluster

    Submit Job

    Record Results

    12

    3Auto-Termina

    te 4

  • 16 Cloudera, Inc. All rights reserved.

    CUSTOMER 360

    : http://blog.godatadriven.com/schiphol-implements-datasciencesuite.html

  • 21 Cloudera, Inc. All rights reserved.

    $120M(130)

  • 22 Cloudera, Inc. All rights reserved.

    : hZp://techspec[ve.net/2015/08/03/how-gopro-is-using-amazon-bmc-and-cloudera-to-kick-everyone-elses-buZ/

  • 26 Cloudera, Inc. All rights reserved.

    FINRA monitors 50B market events per day to build a holis[c picture of US market ac[vity and make real-[me decisions, while saving $10-20M annually

  • 27 Cloudera, Inc. All rights reserved.

    Airbnb improved their overall booking rate through machine learning algorithms and beZer search to more effec[vely match customers with the right rental property

    CUSTOMER 360

  • 34 Cloudera, Inc. All rights reserved.

    Cloudera Director 2.0 & C5.5 Releases

    API Hive on S3 Spark on S3

    Impala on S3 (beta)

    BI/ HA HA/Kerberos

    DB UI

    ALL WORKLOADS: AWS s3a GUI :

  • 35 Cloudera, Inc. All rights reserved.

    Power BI

    Microso> Azure

    Marketplace

    Marketplace Delivers Full cloud deployment; no hardware dependency Start work in

  • 36 Cloudera, Inc. All rights reserved.

    Get Started AWS Reference Guide GCP Reference Guide Download Cloudera Director www.cloudera.com/downloads

    Try It Out Cloudera Live (includes step-by-step tutorial) AWS Quickstart Azure Marketplace

    Resources API Integra[on & Scrip[ng hZps://github.com/cloudera/director-sdk

    hZps://github.com/cloudera/director-scripts

    Addi[onal Cloud Integra[on hZps://github.com/cloudera/director-spi hZps://github.com/cloudera/director-google-plugin

  • 37 Cloudera, Inc. All rights reserved.

    Cloudera on AWS

  • 38 Cloudera, Inc. All rights reserved.

    Cloudera on AWS HDFSHDFS

    S3CPU /

  • 39 Cloudera, Inc. All rights reserved.

    S3

    HDFS

    Hadoop

    ()

    Hadoop

    HDFSHadoop

    HDFS

    EBS IO

    IO EC2/EBS

    OS

  • 40 Cloudera, Inc. All rights reserved.

    MapReduce YARN Spark Hive Pig Crunch

    c3.8xlarge d2.2xlarge i2.2xlarge i2.4xlarge i2.8xlarge r3.8xlarge m2.4xlarge

    c3.8xlarge d2.8xlarge i2.2xlarge i2.4xlarge i2.8xlarge r3.8xlarge

    HBase Solr Impala

    c3.8xlarge d2.2xlarge i2.4xlarge i2.8xlarge r3.8xlarge

    d2.8xlarge i2.4xlarge i2.8xlarge

    CDH d2.2xlarge i2.2xlarge i2.4xlarge

    d2.8xlarge

  • 41 Cloudera, Inc. All rights reserved.

    VPC 11ACL

    Flume

    IP

    GWNAT

    NAT Linux EC2 DC (VPC or Direct Connect)

  • 43 Cloudera, Inc. All rights reserved.

    S3S3distcpHDFSHadoop

    HDFSHadoopS3

  • 44 Cloudera, Inc. All rights reserved.

    S3distcp / ACL HDFS

    HBase

    2AZHiveRDBMSAmazon RDS

    distcpS3HDFS

  • 45 Cloudera, Inc. All rights reserved.

    Hadoop

    Impala

    SparkKafka Flume

    HDFS HBase

  • 46 Cloudera, Inc. All rights reserved.

    S3()

    (Impala)

  • 47 Cloudera, Inc. All rights reserved.

    Impala on S3 C5.5 JOIN: HDFS, HBase, S3 JOIN(SentryACL): :

    DML

    INSERT / LOAD DATA / CREATE TABLE AS SELECT

    HDFS

  • 48 Cloudera, Inc. All rights reserved.

    Thank you