Security implementation on hadoop

1© Cloudera, Inc. All rights reserved.

Security Implementation on Hadoop

Dr. Wei-Chiu Chuang | Software

Engineer


$ whoami

Software Engineer, Cloudera Apache Hadoop Committer/PMC


Unguarded data stores are the victims


Regulatory Compliance

Organizations can be fined up to 4% of annual global turnover for breaching GDPR

or €20 Million


Security Implementation


Disclaimer

This talk serves as a general guideline for

security implementation on Hadoop.

The actual implementation procedures and

scope of implementation vary on a case-

by-case basis, and should be assessed by

Cloudera’s Professional Services team or

certified Cloudera SI Partners.


Non-secure #0Data Free for All


Firewall

ActiveDirectory/KDC

Hadoop cluster

Cloudera Manager

Gateway node

Cloudera NavigatorDatacenter

Applications


High Availability made Easy


Identity Management

Simple AuthenticationFile group ownership• AD integration• SSSD or CentrifyConsideration in large enterprises.

SSSD

via


System Diagram #0

Firewall

ActiveDirectory

Master

Worker Worker Worker

Cloudera Manager

Master

(SSSD/Centrify)


Simple authentication =

no authentication


Minimal Security #1

Reduce Risk Exposure


Kerberos

EXAMPLE.COM

KDC

[email protected]

Hadoop

[email protected]

user

Strong Authentication

KDC

• MIT

• ActiveDirectory (more common)

realmprimary


Kerberos

Consideration in large corporates

Time synchronization

CM Kerberos Wizard

• Configure AD to create a Kerberos

principal for CM server, and to

delegate CM the ability to

create/manage Kerberos principals


LDAP Authentication

* LDAP over SSL


Authorization/Access Control

HDFS File ACL YARN job submission

Hbase ACLs Oozie ACL

Access Control List (ACLs)

Hive

Sentry Managed

(RBAC)

Impala


Auditing


Backup/Disaster Recovery

Cloudera Backup/Disaster Recovery (BDR)

• A high performance data replicator

• Copies incremental data on the source cluster at specified schedules

Supports

Kerberos

Data encryption

HDFS replication to cloud


Kerberized BDR Best Practice

Production DR

Cloudera BDRPROD.EXAMPLE.COM

Cross-realm trustKDC KDC

DR.EXAMPLE.COM


Firewall

System Diagram #1

ActiveDirectory/ KDC

Master


Cloudera Manager

Kerberos

Master

(SSSD/Centrify)

DR


More Security #2

Managed, Secure, Protected


Data In-Transit Encryption

RPC encryption

Data transport encryption

• Supports AES CTR, up to 256-bit

key length

HTTP TLS/SSL encryption

• No self-signed certificates in

production

Master


Master

Application

RPC encryption

Transport encryption

TLS/SSL


Data At-Rest Encryption

Transparent encryption

Supports any Hadoop applications

Encryption Zone

$ hadoop key create mykey

$ hadoop fs -mkdir /zone

$ hdfs crypto -createZone -keyName mykey -path /zone

/

/tmp/zon

e

foo bar

Encryption zone


Key Management Server Deployment (non-prod)

HDFS NameNode

Client

Java Keystore

KMS

Keystorefile

Separation of duties

• Encryption Zone Key (EZK) is stored in

KMS server

• HDFS super user can not decrypt files


Key Management Server/Key Trustee Server Deployment

HDFS NameNode

ClientKey Trustee

KMS

Key Trustee KMS

Firewall

Key Trustee Server

(Active)

Key Trustee Server

(Passive)

synchronization

(or more)


KMS+KTS+HSM Deployment

HDFS NameNode

Client HSM KMS

HSM KMS

Firewall

Key Trustee Server

(Active)

Key Trustee Server

(Passive)

synchronization

Key HSM

(or more)

Key HSM

HSM

HSM


Encryption Performance


Troubleshooting: Encryption Performance Anomaly

• Configuration

• AES-NI Hardware acceleration

• OpenSSL library

• Entropy


Fine Grained Access Control with Apache Sentry


Firewall

System Diagram #2

ActiveDirectory/ KDC

Master


Cloudera Manager

Kerberos

Master

KMSKMS

Firewall

KeyTrusteeKeyTrustee

(SSSD/Centrify)


Most Security #3

Secure Data Vault


Data Redaction

Personal Identifiable Information

• PCI-DSS, HIPAA

Best practice

Password

• stores in credential files, not in configuration

Log, queries

• Cloudera Manager


Full Encryption

Encrypt Data Spills

• MapReduce

• Impala

• Hive

• Flume

OS-level encryption

• Navigator Encrypt


Security Vulnerabilities


Vulnerability Response and Process

Vulnerability reports

Upstream

Internal

External

Fix Publish

CVE

Cloudera TSB


Cloudera Certified Technology


Cloudera Certified Technology Partners

Data Sources Data IngestProcess, Refine

& PrepData Discovery Advanced Analytics

Connected Machines/Data sources

Other Data Sources


A certified product ensures it integrates with a secure cluster

• Authenticate via Kerberos or LDAP

Authentication

• Handle Apache Sentry with Hive, Impala, Search, HDFS

Authorization

• Support HDFS transport encryption, at-rest encryption; support SSL/TLS connection encryption

Encryption


Cloudera SDX


Cloudera Enterprise

42

The modern platform for machine learning and analytics optimized for the cloud

EXTENSIBLE SERVICES

CORE SERVICESDATA

ENGINEERINGOPERATIONAL

DATABASEANALYTIC DATABASE

DATA CATALOG

INGEST & REPLICATION

SECURITY GOVERNANCEWORKLOAD

MANAGEMENT

DATA SCIENCE

S3 ADLS HDFS KUDUSTORAGESERVICES


• Unified security – protects sensitive data with consistent

controls, even for transient and recurring workloads

• Consistent governance – enables secure self-service access

to all relevant data and increases compliance

• Easy workload management – increases user productivity

and boosts job predictability

• Flexible ingest and replication – aggregates a single copy of

all data, provides disaster recovery, and eases migration

• Shared catalog – defines and preserves structure and

business context of data for new applications and partner

solutions

Open platform servicesBuilt for multi-function analytics | Optimized for cloud


Successful use cases


Cloudera Overview & Financial Services Focus

2000Strong Partner

Ecosystem

+

1600 Employees Globally

+

19 Of the 30 G-SIBs Run on Cloudera

Strong Focus & Momentum in Financial Services

3 Of the Fortune 500

Top 5 Insurers Run on Cloudera

5 Of the Top 6 Asset Management Firms

Run on Cloudera

200+ Financial Services Customers


Building a Fantastic Customer Experience

• Improved customer experience• 80 percent reduction in operating costs

through a wide-range of customer service and operational improvements

• Decrease in cost to service customers while increasing revenue through better service

CUSTOMER 360

FINANCIAL SERVICES» PREDICTIVE ANALYTICS» 360 CUSTOMER VIEW» OPERATIONAL ANALYTICS


Large healthcare provider enables practitioners to recommend at-home actions to prevent hospital visits

• Flexible, automatic data classification for diverse medical ontologies

• Self-service data discovery for real-time, data-driven decisions


Thank you

Wei-Chiu Chuang | [email protected]


More information on Hadoop Security


Books authored by Clouderans

Engineering

Security implementation on hadoop