Nifi workshop

Introducing Apache Nifi

Yifeng Jiang Solutions Engineer, Hortonworks

© Hortonworks Inc. 2011 – 2015. All Rights Reserved

About Me

Yifeng Jiang•  Solutions Engineer, Hortonworks•  Apache HBase book author•  I like hiking•  Twitter: @uprush

Page 3 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Agenda

•  Introduction to Nifi •  Nifi Demo •  Nifi Use Case


Introduction to Apache NiFi


Nifi Overview

Nifi is an easy to use, powerful, and reliable system to process and distribute data.


NiFi Terminology FlowFile

•  Unit of data moving through the system •  Content + Attributes (key/value pairs)

Processor •  Performs the work, can access FlowFiles

Connection •  Links between processors •  Queues that can be dynamically prioritized

Process Group •  Set of processors and their connections •  Receive data via input ports, send data via output ports


NiFi - User Interface

•  Drag and drop processors to build a flow •  Start, stop, and configure components in real time •  View errors and corresponding error messages •  View statistics and health of data flow •  Create templates of common processor & connections


NiFi - Provenance

•  Tracks data at each point as it flows through the system

•  Records, indexes, and makes events available for display

•  Handles fan-in/fan-out, i.e. merging and splitting data

•  View attributes and content at given points in time


NiFi - Queue Prioritization

•  Configure a prioritizer per connection

•  Determine what is important for your data – time based, arrival order, importance of a data set

•  Funnel many connections down to a single connection to prioritize across data sets

•  Develop your own prioritizer if needed


NiFi - Architecture


Nifi Cluster

•  Nifi Cluster Manager •  Nifi Cluster Nodes

•  Primary Node

•  Isolated Processor OS/Host

JVM

Flow Controller

Web Server

Processor 1 Extension N

FlowFile Repository

Content Repository

Provenance Repository

Local Storage

OS/Host

JVM

Flow Controller

Web Server


FlowFile Repository

Content Repository


Local Storage

OS/Host

JVM

NiFi Cluster Manager – Request Replicator

Web Server

Master NiFi Cluster Manager (NCM)

OS/Host

JVM

Flow Controller

Web Server


FlowFile Repository

Content Repository


Local Storage

Slaves NiFi Nodes


NiFi Demo


Nifi Demo •  The demo cluster: ambari deployment, NCM

•  Real-time indexing in Solr & Banana

•  Nifi UI •  Flow statistics •  Data provenance, event details, replay

•  Add a Processor to push data to Kafka

•  Nifi data on the node •  Flow file repository •  Content repository

•  Provenance repository


Site to Site -- Flow

Nifi Cluster A (source)

Nifi Cluster B (destination)

Site to site

Remote Process Group

Flow file attributes transferred


Site to Site – Data Provenance

Nifi Cluster A (source)

Nifi Cluster B (destination)

Event details at cluster B


NiFi Use Cases


Use Cases – Index JSON 1.  Pull in Tweets using Twitter API

2.  Extract language and text into FlowFile attributes

3.  Get non-empty English tweets ${twitter.text:isEmpty():not():and(

${twitter.lang:equals("en")})}

4.  Merge together JSON documents based on quantity, or time

5.  Use dynamic field mappings to select fields for indexing:


Use Cases – Index a Relational Database 1.  GenerateFlowFile acts a timer to trigger

ExecuteSQL (Future plans to not require in an incoming FlowFile to ExecuteSQL NIFI-932)

2.  ExecuteSQL performs a SQL query and streams the results as an Avro datafile Use expression language to construct a dynamic date range:

${now():toNumber():minus(60000)

:format(‘YYYY-MM-DD’}

3.  Convert Avro to JSON using built in ConvertAvroToJSON processor

4.  Stream JSON update to Solr


Built-in Processors

•  90 built-in processors •  Well-defined API

•  Easy to implement



Thank You

Software

Nifi workshop