Improvements in Hadoop Security

© Hortonworks Inc. 2011

Improvements in Hadoop Security

Sanjay Radia

[email protected]

@srr

Chris Nauroth

[email protected]

@cnauroth

Page 1


Hello

Sanjay Radia

• Founder, Hortonworks

• Part of the Hadoop team at Yahoo! since 2007

– Chief Architect of Hadoop Core at Yahoo!

– Long time Apache Hadoop PMC and Committer– Designed and developed several key Hadoop features

• Prior

– Data center automation, virtualization, Java, HA, OSs, File Systems (Startup, Sun Microsystems, …)

– Ph.D., University of Waterloo

Chris Nauroth

• Member of Technical Staff, Hortonworks

– Apache Hadoop Committer

– Major contributor to HDFS ACLs

• Hadoop user since 2010

– Prior employment experience deploying, maintaining and using Hadoop clusters

Page 2Architecting the Future of Big Data


Overview

• Models of Deployment

– Secure and insecure

• Hadoop Authentication

– The how and why

– Knox – perimeter security

• Authorization – existing and what is new

– HDFS

– Tables and Hive

– HBase and Accumulo

• Data protection and encryption

– Wire

– Data at rest



Two Reasons for Security in Hadoop

Hadoop Contains Sensitive Data

– As Hadoop adoption grows so too has the types of data organizations look to store. Often the

data is proprietary or personal and it must be protected.

– In this context, Hadoop is governed by the same security requirements as any data center

platform.

Hadoop is subject to Compliance adherence

– Organizations are often subject to comply with regulations such as HIPPA, PCI DSS, FISAM that

require protection of personal information.

– Adherence to other Corporate security policies.

1

2


Three Models of Hadoop Deployment

• Insecure cluster

– You have protected via the perimeter

– You trust the code that runs in the system

– - note In Hadoop cluster, user submitted code runs inside the cluster

– (Note true in typical client-server applications)

– The client side libraries pass the client’s login credential

– There is not end-end-authentication here

– Authorization is done against this credentials

• Secure cluster

– Full authentication

– Can run arbitrary code in jobs

• Perimeter security using Knox

– Internal cluster can be secure or insecure depending on your needs


Pillars of Hadoop Security

Authorization

Restrict access to

explicit data

Audit

Understand who did

what

Data Protection

Encrypt data at rest &

motion

AD/Kerberos in native

Apache Hadoop

Perimeter Security

with Apache Knox

Gateway

Authentication

Who am I/prove it?

Control access to cluster.


Hadoop Authentication Overview

• Kerberos/Active Directory based security

– SSO – users do not have to re-login into Hadoop

– Hadoop accounts do not have to be recreated

– Caveat – MR Task isolation require Unix accounts for each user current but this is going away with Linux containers

– Hadoop tokens – supplement the Kerberos authentication

– Delegation tokens – deal with the delayed job execution

– Block tokens – capabilities to deal with the distributed nature of HDFS

– Trusted Proxies – support for third party services to act as proxy

– Oozie

– Gateways – HDFS proxy, Knox, etc

• Base security infrastructure being extended to support other authentication services besides

Kerberos

• Knox – Perimeter Security and Rest Gateway

Page 7


User DB/Account

Single-signon using your organization’s existing user DB

• No need to create Hadoop user accounts to be created

– Note the OS account on compute nodes are for isolation – this will be removed when we use Linux containers

• Currently supports Kerberos (and LDAP indirectly)

• The RPC layer’s authentication is fairly general

– Supports Kerberos and Hadoop’s tokens

– Can be extended


Why the tokens

• Why does Hadoop have its own tokens?

– Standard client-server security model is not sufficient for Hadoop

– Works when logged-in client is directly accessing a Hadoop service

– But for a job, the execution happens much later

– The job submitter has long logged off

• Hence we needed to add delegation tokens

• HDFS is a distributed service and needed to add capability-like tokens for datanode

authentication

The permissions/ ACLs are in the Namenode


Apache KnoxPerimeter Security for Hadoop REST APIs

Architecting the Future of Big DataPage 10


The Gateway or Edge Node

• Hadoop APIs can be used from any desktop after SSO login

– FileSystem and MapReduce Java APIs

– Pig, Hive and Oozie clients (that wrap the Java APIs)

• However it is typical to Use “Edge Node” or “Gateway Node” that is “inside” cluster

– The libraries for the APIs are generally only installed on the gateway

– Users SSH to Edge Node and execute API commands from shell

Page 11

HadoopUser Edge Node

SSH


• Single Hadoop access point

• REST API hierarchy

• Consolidated API calls

• Multi-cluster support

• Eliminates SSH “edge

node”

• Central API management

• Central audit control

• Simple Service level

Authorization

• SSO Integration –

Siteminder, API Key*,

OAuth* & SAML*

• LDAP & AD integration

Perimeter Security with Apache Knox

Integrated with existing

systems to simplify identity

maintenance

Incubated and led by Hortonworks,

Apache Knox provides a simple and open framework for

Hadoop perimeter security.

Single, simple point of

access for a cluster

Central controls ensure

consistency across one or

more clusters


Hadoop REST APIs

• Useful for connecting to Hadoop from the outside the cluster

• When more client language flexibility is required

– i.e. Java binding not an option

• Challenges (Knox addresses these challenges)

– Client must have knowledge of cluster topology

– Required to open ports (and in some cases, on every host) outside the cluster

Page 13

Service API

WebHDFS Supports HDFS user operations including reading files, writing to files,

making directories, changing permissions and renaming. Learn more about

WebHDFS.

WebHCat Job control for MapReduce, Pig and Hive jobs, and HCatalog DDL

commands. Learn more about WebHCat.

Hive Hive REST API operations

HBase HBase REST API operations

Oozie Job submission and management, and Oozie administration. Learn more

about Oozie.

http://hadoop.apache.org/docs/r1.0.4/webhdfs.html

http://people.apache.org/~thejas/templeton_doc_latest/installation.html

http://oozie.apache.org/


What can be done today?

Authorization

Restrict access to

explicit data

Audit

Understand who did

what

Data Protection


motion

Previously• All Services: Service level ACLs

• HDFS: Permissions

• Yarn: Queue ACLs

• Hive/Pig Tables: Table level via

HDFS

• Apache Accumulo: Cell level

• HBase: Namespace, Table,

Column Family and Column level

ACLs

Authentication

Who am I/prove it?


Hadoop 2.x• HDFS: ACLs

• Hive: Column level ACLs

• HBase: Cell level ACLs

• Knox:

• Rest Service level Authorization

• Access Audit with Knox


HDFS ACLs

• Existing HDFS POSIX permissions good, but not flexible enough

– Permission requirements may differ from the natural organizational hierarchy of users and groups.

• HDFS ACLs augment the existing HDFS POSIX permissions model by implementing the POSIX

ACL model.

– An ACL (Access Control List) provides a way to set different permissions for specific named users or named

groups, not only the file’s owner and file’s group.



HDFS File Permissions Example

• Authorization requirements:

– In a sales department, they would like a single user Maya (Department Manager) to

control all modifications to sales data

–Other members of sales department need to view the data, but can’t modify it.

–Everyone else in the company must not be allowed to view the data.

• Can be implemented via the following:

Read/Write perm for user

mayaUser

GroupRead perm for group sales

File with sales data


HDFS ACLs

• Problem

–No longer feasible for Maya to control all modifications to the file

– New Requirement: Maya, Diane and Clark are allowed to make modifications

– New Requirement: New group called executives should be able to read the sales data

–Current permissions model only allows permissions at 1 group and 1 user

• Solution: HDFS ACLs

–Now assign different permissions to different users and groups

Owner

Group

Others

HDFS

Directory

… rwx

… rwx

… rwx

Group D … rwx

Group F … rwx

User Y … rwx


HDFS ACLs

New Tools for ACL Management (setfacl, getfacl)

–hdfs dfs -setfacl -m group:execs:r-- /sales-data

–hdfs dfs -getfacl /sales-data # file: /sales-data # owner: maya # group:

sales user::rw- group::r-- group:execs:r-- mask::r-- other::--

– How do you know if a directory has ACLs set?

–hdfs dfs -ls /sales-data Found 1 items -rw-r-----+ 3 maya sales 0

2014-03-04 16:31 /sales-data


HDFS ACLs

Default ACLs–hdfs dfs -setfacl -m default:group:execs:r-x /monthly-sales-data

–hdfs dfs -mkdir /monthly-sales-data/JAN

–hdfs dfs –getfacl /monthly-sales-data/JAN

– # file: /monthly-sales-data/JAN # owner: maya # group: sales user::rwx group::r-

x group:execs:r-x mask::r-x other::--- default:user::rwx default:group::r-

x default:group:execs:r-x default:mask::r-x default:other::---


HDFS ACLs Best Practices

• Start with traditional HDFS permissions to implement most permission requirements.

• Define a smaller number of ACLs to handle exceptional cases.

• A file with an ACL incurs an additional cost in memory in the NameNode compared to a file that

has only traditional permissions.



Tables and Hive



Table ACLs – The Challenge and Solution

• Hive and Pig have traditionally offer full table access control via HDFS access control

• The challenge in column-level access control

– Hive and Pig queries are executed as Tez-based tasks that access the HDFS files directly

– HDFS does not have knowledge of columns (there are several file/table formats)

• Solution for Column level ACLs

– Let Hive server check and submit the query execution

– Let the table be accessible only by special user (“HiveServer”)

– But one has to restrict the UDFs and file formats

– Good news: Hive provides an authorization plugin to do this cleanly

• Use standard SQL permission constructs

– GRANT/REVOKE

• Store the ACLs in Hive Metastore instead of some external DB

• But what about Pig, there is no Pig server …



Hive ATZ-NG – Architecture

HDFS

Metastore

HiveServer2

O/JDBC Beeline CLI

• ATZ-NG is called for O/JDBC & Beeline CLI

• Standard SQL GRANT / REVOKE for management

• Privilege to register UDF restricted to Admin user

• Policy integrated with Table/View life cycle

Storage Based Authorization Provider

Hive

CLI

OozieHue

PIG HCat

Ambari

0. Enable HiveATZ-NG

1. Authentication

UDFs

Protected – column level

Protected – table level

Restrict direct access to Metastore

Protect HDFS with Kerberos & HDFS ACL

ATZ-NG2. Authorization


What about MR/Pig

• Note there is no Pig/MR server to submit and check column ACLs

• Hence in the same cluster running Hive

–You can cannot give Pig similar access control

–If Pig/MR is important,

–Use coarse grained table level

–Or run Pig/MR as privileged uses with full table level access

Page 26


Hive ATZ-NG Example

Page 28


Scenario

• Objective: Share Product Management Roadmap securely

• Actors:

–Admin Role – Specified in hive-site

– Admin role controls role memberships

–Product Management Role

– Should be able to create, read all road map details.

– Members: Vinay Shukla, Tim Hall

–Engineering Role

– Should be able to read (see) all roadmap details

– Members: Kevin Minder, Larry McCay

Page 29


Step 1: Admin role Creates Roles, Adds Users

1. CREATE ROLE PM;

1. CREATE ROLE ENG;

1. GRANT ROLE PM to user timhall with admin option;

1. GRANT ROLE PM to user vinayshukla;

1. GRANT ROLE ENG to user kevinminder with admin option;

1. GRANT ROLE ENG to user larrymccay;


Step 2: Super-user Creates Tables/Views

create table hdp_hadoop_plans (

id int,

hadoop_roadmap string,

hdp_roadmap string

);


Step 3: Users or Roles Assigned To Tables

1. GRANT ALL ON hdp_hadoop_plans TO ROLE PM;

1. GRANT SELECT ON hdp_hadoop_plans TO ROLE ENG;


HBase Cell Level Authorization

• The HBase permissions model already supports ACLs defined at the namespace, table, column

family and column level.

– This is sufficient to meet many requirements

– This can be insufficient if a data model requires protection on individual rows/cells.

– Example: Medical data, each row representing a patient, may require customizing who can see an individual patient’s data,

and the social security number of each row may need further restriction.

Page 33



• Cell level authorization augments the permissions model by allowing ACLs specified on

individual cells.

– ACLs are now supported at the individual cell level.

– Individual operations may choose order of evaluation. Cell level ACLs may be evaluated last or first.

– Evaluating last is useful if the common case is access granted through table or column family ACLs, and cell level

ACLs define exceptions for denial.

– Evaluating first is useful if many users are granted access through cell level ACLs.




• Visibility labels

– Visibility expressions can be stored as metadata in a cell’s tag.

– A visibility expression consists of labels combined with boolean operators.

– E.g. (financial | strategy | research) & !newhire

– This means that a user must be labeled financial or strategy or research and not be a newhire in order to see the column.

– The mapping of users to their labels is pluggable. By default, a user’s labels are specified as authorizations in the

individual operation.

– HBase visibility labels were inspired by similar features in Apache Accumulo, and the model will look very familiar

to Accumulo users.



What can be done today?

Authorization

Restrict access to

explicit data

Audit

Understand who did

what

Data Protection


motion

Wire encryption• In native Hadoop

• With Knox

• SSL for Rest (2.x)

File encryption• Via MR file format

• 3rd Party encryption

tools for col level

encryption

• Native HDFS support

coming

Authentication

Who am I/prove it?



Wire Encryption – for data in motion

Page 37

• Hadoop client to DataNode is via Data Transfer Protocol

– HDFS client reads/writes to HDFS service over encrypted channel

– Configurable encryption strength

• ODBC/JDBC Client to HiveServer 2

– Encryption is via SASL Quality Of Protection

• Map to Reduce via shuffle

– Shuffle is over HTTP(S)

– Supports mutual authentication via SSL

– Host name verification enabled

• Rest Protocols

– SSL support


Data at Rest

• Coming: HDFS encrypted file system currently under development in Apache

– https://issues.apache.org/jira/browse/HADOOP-10150

– https://issues.apache.org/jira/browse/HDFS-6134



XA SecureA Major step forward in Hadoop security

See Shaun Connolly’s Key note on Wednesday June 4



Security in Hadoop with HDP + XA Secure

Authorization

Restrict access to

explicit data

Audit

Understand who

did what

Data Protection

Encrypt data at

rest & in motion

• Kerberos in native

Apache Hadoop

• HTTP/REST API

Secured with

Apache Knox

Gateway

• MapReduce Access Control Lists

• HDFS Permissions, HDFS ACL,

• Audit logs in with HDFS & MR

• Hive ATZ-NG

• Cell level access control in

Apache Accumulo

Authentication

Who am I/prove it?

• Wire encryption

in Hadoop

• Orchestrated

encryption with

3rd party tools

• HDFS, Hive and

Hbase

• Fine grain

access control

• RBAC

• Centralized

audit reporting

• Policy and

access history

• Future roadmap

• Strategy to be

finalized

HD

P 2

.1X

AS

ecure

Centralized Security Administration

• As-Is, works with

current

authentication

methods


Open Source?

•Yes XASecure technology will be open sourced

–Not just a Apache license where you are forced to get

the latest from Hortonworks

–But a full-fledged Apache Project that is truly open to the

community of developers and users

•See Shaun Connolly’s Keynote on Wednesday for

details



Summary

• Very strong Authentication via Kerberos and Active directory

– Uses your organization's user DB and integrates to its group and role membership

– Supplemented by Hadoop tokens

– Note these are necessary due to delayed job execution after user logs-off

• Strong fine grained authentication with some recent improvements

– HDFS ACLs

– Hive – integrated via SQL model and Hive Metastore

– Note a hacked side addon

– HBase Cell Level Authorization

• Strong encryption support

– Wire

– Data

– Some improvements coming soon

• Every product has audit logs

• XASecure adds a major step forward

– Yes it will be open sourced as a Apache Project



Thank you, Q&A

Page 43

Resource Location

Hortonworks Security Labs http://hortonworks.com/labs/security/

Apache Knox Project Page http://knox.incubator.apache.org/

HDFS ACLs Blog Post http://hortonworks.com/blog/hdfs-acls-fine-grained-permissions-hdfs-files-hadoop/

Encrypted File System

Development

https://issues.apache.org/jira/browse/HADOOP-10150

https://issues.apache.org/jira/browse/HDFS-6134

HBase Cell Level

Authorization

https://blogs.apache.org/hbase/entry/hbase_cell_security

Learn more

Software

Improvements in Hadoop Security