1 © Hortonworks Inc. 2011–2018. All rights reserved

27-29 November, Vilnius

Apache Metron in the Real WorldDave Russell - Hortonworks

www.roaringelephant.org

2 © Hortonworks Inc. 2011–2018. All rights reserved

Who am I?

3 © Hortonworks Inc. 2011–2018. All rights reserved

4 © Hortonworks Inc. 2011–2018. All rights reserved

Why Apache Metron?

5 © Hortonworks Inc. 2011–2018. All rights reserved

Months until breach noticed

Avg. months log retention

9 6

VS

3Months missing

6 © Hortonworks Inc. 2011–2018. All rights reserved

28 Months

Police One/Berkut

Yahoo/FSB FB/Cambridge Analytica

35 Months 48 Months

Time until breach actually noticed

7 © Hortonworks Inc. 2011–2018. All rights reserved

“Sometime in the next few years we're going to have our first

category-one cyber-incident; one that will need a national response.”

Ian LevyTechnical Director

National Cyber Security Centre

8 © Hortonworks Inc. 2011–2018. All rights reserved

Andhra Pradesh Police, IndiaAristotle University of Thessaloniki, GreeceAutomobile Dacia, RomaniaCambrian College, CanadaChinese public security bureauCJ CGVDalian Maritime UniversityDeutsche BahnDharmais Hospital, IndonesiaFaculty Hospital, Nitra, SlovakiaFedExGarena Blade and SoulGuilin University Of Aerospace TechnologyGuilin University Of Electronic TechnologyHarapan Kita Hospital[disambiguation needed], IndonesiaHezhou University

SandvikSão Paulo Court of JusticeSaudi Telecom CompanySberbankShandong UniversityState Governments of India Government of GujaratGovernment of KeralaGovernment of MaharashtraGovernment of West BengalSuzhou Vehicle AdministrationSun Yat-sen University, ChinaTelefónicaTelenor Hungary, HungaryTelkom (South Africa)Timrå Municipality, SwedenUniversitas Jember, IndonesiaUniversity of Milano-Bicocca, ItalyUniversity of Montreal, CanadaVivo, Brazil

HitachiHondaInstituto Nacional de Salud, ColombiaLakeridge HealthLAKSLATAM Airlines GroupMegaFonMinistry of Internal Affairs of the Russian FederationMinistry of Foreign Affairs (Romania)National Health Service (England)NHS ScotlandNissan Motor Manufacturing UKO2, GermanyPetrobrásPetroChinaPortugal TelecomPulse FMQ-ParkRenaultRussian Railways

9 © Hortonworks Inc. 2011–2018. All rights reserved

2018 so far...

340M Records

150M Records

92M RecordsAnd many, many, many more..https://en.wikipedia.org/wiki/List_of_data_breaches

10 © Hortonworks Inc. 2011–2018. All rights reserved

What Does Apache Metron Look Like?

11 © Hortonworks Inc. 2011–2018. All rights reserved

Security telemetry source: authentication logsSecurity telemetry source: authentication logs

12 © Hortonworks Inc. 2011–2018. All rights reserved

13 © Hortonworks Inc. 2011–2018. All rights reserved

14 © Hortonworks Inc. 2011–2018. All rights reserved

15 © Hortonworks Inc. 2011–2018. All rights reserved

What is Apache Metron?

16 © Hortonworks Inc. 2011–2018. All rights reserved

Built on top on proven open source big data technology

17 © Hortonworks Inc. 2011–2018. All rights reserved

An architecture for real-time cybersecurity analytics

18 © Hortonworks Inc. 2011–2018. All rights reserved

Telemetry Data Source

19 © Hortonworks Inc. 2011–2018. All rights reserved

Telemetry Data Collectors

20 © Hortonworks Inc. 2011–2018. All rights reserved

Cyber Security Stream Processing Pipeline

21 © Hortonworks Inc. 2011–2018. All rights reserved

Profiling by time

t = 1 t = 2 t = 3 t = n

Wide range of algorithms including:à HyperLogLogPlusà Bloom filtersà T-digestsà Statistical Baseliningà Hashing functionsà Outlier detectionà GeoHashing over timeà Locality Sensitive HashingApprox. Data

SketchApprox. Data

SketchApprox. Data

SketchApprox. Data

Sketch

Combined Baseline

Statistic

22 © Hortonworks Inc. 2011–2018. All rights reserved

Cyber Security Stream Processing Pipeline

23 © Hortonworks Inc. 2011–2018. All rights reserved

Apache Metron Modules

24 © Hortonworks Inc. 2011–2018. All rights reserved

Who is Using Apache Metron (Part 1)

25 © Hortonworks Inc. 2011–2018. All rights reserved

26 © Hortonworks Inc. 2011–2018. All rights reserved

27 © Hortonworks Inc. 2011–2018. All rights reserved

28 © Hortonworks Inc. 2011–2018. All rights reserved

The Wider Apache Metron Ecosystem

29 © Hortonworks Inc. 2011–2018. All rights reserved

30 © Hortonworks Inc. 2011–2018. All rights reserved

Who is Using Apache Metron (Part 2)

31 © Hortonworks Inc. 2011–2018. All rights reserved

32 © Hortonworks Inc. 2011–2018. All rights reserved

33 © Hortonworks Inc. 2011–2018. All rights reserved

34 © Hortonworks Inc. 2011–2018. All rights reserved

35 © Hortonworks Inc. 2011–2018. All rights reserved

Deploying Apache Metron

36 © Hortonworks Inc. 2011–2018. All rights reserved

AD/AssetDB/HR/Threat

HDF

HDFS

NiFi - Ingest

HDP

Phase 0 – Current State

ADP Event Broker (Kafka)

ADP Smart Connectors

ADP Logger

ArcSight ESM

Security Assets

3

1

2

4

5

37 © Hortonworks Inc. 2011–2018. All rights reserved

HDF

Zeppelin

HDFS

NiFi - Ingest

Kafka MQ

Storm Parse / Enrich / GeoIP / Index

SolrInvestigator UI

HDP

Phase 1 - Ingest and Archive

ADP Event Broker (Kafka)

ADP Smart Connectors

ADP Logger

ArcSight ESM

Security Assets AD/AssetDB/HR/Threat

Spark

Historical Analysis

10

6

87

9

11

12

13

Banana

38 © Hortonworks Inc. 2011–2018. All rights reserved

HDF

Zeppelin

HDFS

NiFi - Ingest

Kafka MQ

Storm Parse / Enrich / GeoIP / Index

Solr

Enrichment Data

Investigator UI

HDP

Phase 2 – Enrich and Threat Intel

ADP Event Broker (Kafka)

ADP Smart Connectors

ADP Logger

ArcSight ESM

Security Assets AD/AssetDB/HR/Threat

Spark

Historical Analysis

14

Banana / Kibana / ZoomData

39 © Hortonworks Inc. 2011–2018. All rights reserved

HDF

Zeppelin

HDFS

NiFi - Ingest

Kafka MQ

Storm Parse / Enrich / GeoIP / Index

Solr

Enrichment Data

Metron Profiler

Triage

Alert

Investigator UI

HDP

Phase 3 – NiFi Data Ingestion + Analytics / UEBA Profiling

ADP Event Broker (Kafka)

ADP Smart Connectors

ADP Logger

ArcSight ESM

Security Assets AD/AssetDB/HR/Threat

Spark

Historical Analysis

Source Data (via NiFi)

15

16

Banana

40 © Hortonworks Inc. 2011–2018. All rights reserved

HDF

Source Data (via NiFi)

Zeppelin

HDFS

Spark

Historical Analysis

Model as a Service

NiFi - Ingest

Kafka MQ

Storm Parse / Enrich / GeoIP / Index

Automated Response

Solr

Enrichment Data

Netflow / PCAP /

Snort (Kafka direct)

Metron Profiler

Triage

Alert

Investigator UI

HDP

Phase 4 – ArcSight Logger Migration + New Data Sources

ADP Event Broker (Kafka)

ADP Smart

Connectors

ADP Logger

ArcSight ESM

Security Assets

Banana

AD/AssetDB/HR/Threat

17

18

19

20

41 © Hortonworks Inc. 2011–2018. All rights reserved

Considerations for Sizing Apache Metron

42 © Hortonworks Inc. 2011–2018. All rights reserved

• Events per second (average and peak)• Retention time for Hot / Warm / Cold

zones• Enrichments• Node sizing• I/O Considerations• PCAP?

Sizing an HCP deployment

43 © Hortonworks Inc. 2011–2018. All rights reserved

3 Months

Hot

Warm

Fast indexed layer (Solr / ES) ~3 months

Warm HDFS layer ~3 months

44 © Hortonworks Inc. 2011–2018. All rights reserved

12 Months

Hot

Warm

Fast indexed layer (Solr / ES) ~3 months

Warm HDFS layer ~12 months

45 © Hortonworks Inc. 2011–2018. All rights reserved

Hot

Warm Cold

Fast indexed layer (Solr / ES) ~3 months

Warm HDFS layer ~12 months

Cold HDFS layer +12 months

24 Months

46 © Hortonworks Inc. 2011–2018. All rights reserved

Cold

Beyond 24 months

Hot

Warm ColdColdColdCold

Fast indexed layer (Solr / ES) ~3 months

Warm HDFS layer ~12 months

Cold HDFS layer +12 months

47 © Hortonworks Inc. 2011–2018. All rights reserved

Questions?

48 © Hortonworks Inc. 2011–2018. All rights reserved

49 © Hortonworks Inc. 2011–2018. All rights reserved

Appendix