1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Security & Governance using

Apache Ranger & Apache Atlas

October 2016

Madhan NeethirajDirector - Engineering, Security & Governance

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Disclaimer

This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed.

Project capabilities are based on information that is publicly available within the Apache Software Foundation project websites ("Apache"). Progress of the project capabilities can be tracked from inception to release through Apache, however, technical feasibility, market demand, user feedback and the overarching Apache Software Foundation community development process can all effect timing and final delivery.

This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product.

Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Since this document contains an outline of general product development plans, customers should not rely upon it when making purchasing decisions.

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Agenda

• Introduction• Apache Ranger

• Overview• Authorization policies• Row-filter, Column-masking policies• Audit logs

• Apache Atlas• Overview• Lineage• Classification

• Demo• Q & A

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Ranger: Overview

Centralized authorization and auditing across Hadoop components• HDFS, Hive, HBase, Knox, Strom, YARN, Kafka, Solr, ..• Audit logs to: Solr, HDFS, Log4j, ..

Access Authorization based on Resources, Resource Classification• Policies for specific set of resources – like a Hive database/table/column• Policies for resource classifications – like PII, PHI, PCI

Row-filter, Column-masking based on policies• Restrict the rows accessible in a table based on users/groups/runtime-context

• example: restrict users to access customer records for specific regions only• Mask or anonymize sensitive columns based on users/groups/runtime-context

• example: only last 4 digits of account number should be available to few user-groupsExtensible Architecture

• Custom policy conditions, context enrichers• Easy to enable Ranger authorization and auditing for new components

Encryption keys management to support Transparent Data Encryption

5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Ranger: Centralized Administration

Single pane of glass for security administration across multiple Hadoop components

6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Ranger: Authorization PoliciesConsistent authorization policy structure across Hadoop components

HDFS Resources

Users/Groups/Permissions

Hive Resources

Users/Groups/Permissions

7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Ranger: Row-filter, Column-masking Policies

Row Filter to apply

Mask to apply

8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Ranger: Tag-based Policies

Pick the tag

Deny access to data after expiry date with the exception of ‘admin’ user

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Ranger: Access Audit Logs• Apache Ranger Plugins generate detailed audit logs of access to protected resources• Audit logs to multiple destinations: Solr, HDFS, Log4j appender• Interactive view of audit logs in Apache Ranger admin console

10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Ranger: Architecture

HBase

Ranger Administration Portal

HDFS

Hive Server2

Ranger Audit Server

Ranger Plugin

Had

oop

Com

pone

nts

Ent

erpr

ise

Use

rs

Legacy Tools and Data Governance

Knox

Ranger Policy Server

Storm

Solr

HDFS

Ranger Plugin

Ranger Plugin

Ranger Plugin

Ranger Plugin

Solr

YARN

Kafka

Ranger Plugin

NiFi

Atlas

Ranger Plugin

Ranger Plugin

Ranger Plugin

Ranger Plugin

Ranger UgSync

Ranger TagSync

LDAP/AD/OS

Atlas

11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Atlas: Introduction

Metadata Repository• Flexible type system to capture schema/metadata of multiple components• Out-of-box models for Hive, HDFS, Storm, Falcon, Sqoop

Data Lineage/Provenance• Captures data lineage across components

Classification• Use tags to classify the data – like PII, PHI, PCI, EXPIRES_ON• Support for attributes in tags – like expiry_date

Search• Search using classifications, attributes• Advanced search using DSL; convenient full-text search

Integrations• With Apache Hive, Apache Storm, Apache Falcon, Apache Sqoop for metadata and lineage• With Apache Ranger for classification based security

APIs to add support for more components

12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Atlas: Lineage

13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Atlas: Classification

14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Atlas: Architecture

15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Demo

16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Questions

17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

References

18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

References

• Apache Atlas• http://atlas.apache.org• http://hortonworks.com/apache/atlas

• Apache Ranger• http://ranger.apache.org• http://hortonworks.com/apache/ranger

• Apache Ranger wiki• https://cwiki.apache.org/confluence/display/RANGER

• Tag based policies• https://cwiki.apache.org/confluence/display/RANGER/Tag+Based+Policies

• Row-filtering and column-masking policies• https://

cwiki.apache.org/confluence/display/RANGER/Row-level+filtering+and+column-masking+using+Apache+Ranger+policies+in+Apache+Hive