Hortonworks Data in Motion Webinar Series - Part 6, Edge Intelligence IoT MiniFi

Harnessing Data-in-Motion with Hortonworks DataFlow

Edge intelligence for IoT with Apache MiNiFi (subproject of Apache NiFi)

Joe PercivallSoftware Engineer

Anna YongProduct Marketing

Agenda• Apache NiFi Refresher

• Intro to Apache MiNiFi

• Demo

• Next Step

• How MiNiFi can be used – use cases

EMBRACE AN OPEN APPROACH

MASTER THE VALUE OF DATA

EVERY BUSINESS IS A DATA BUSINESS

Constrained High-latency Localized context

Hybrid – cloud / on-premises Low-latency Global context

CoreInfrastructure

Harnessing Data in Motion

RegionalInfrastructureSources

Hortonworks DataFlow Manages Data in Motion

CoreInfrastructureSources

Constrained High-latency Localized context

Hybrid – cloud / on-premises Low-latency Global context

RegionalInfrastructure

Requirements for Data in Motion

Perishable Insights

ConnectivityPrioritization

Security

AdaptabilityExtensibility

Scalability

Provenance

Real-Time

Connecting Data Between Ecosystems Without Coding: 170+ Processors

Extract

Duplicate

GeoEnrich

Replace

ConvertSplit

Translate

Route Content

Route Context

Route Text

Control Rate

Distribute Load

Generate Table Fetch

Jolt Transform JSON

Prioritized Delivery

Encrypt

Evaluate

Execute

Syslog

All Apache project logos are trademarks of the ASF and the respective projects.

Apache NiFi

Guaranteed delivery Data buffering

‒ Backpressure‒ Pressure release

Prioritized queuing Flow specific QoS

‒ Latency vs. throughput‒ Loss tolerance

Data provenance

Recovery / recording a rolling log of fine-grained history

Designed for extension Visual command and control Flow templates Policy based security Clustering

Key Features

• Demo

• Next Step

Simplified Ingestion ExampleLet’s consider the needs of a courier service

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster

Distribution Center Core Data Center at HQ

Server Cluster

On Delivery Routes

Trucks Deliverers

Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/Deliverer: Rigo Peter, https://thenounproject.com/rigo/Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/

Courier service from the perspective of NiFi

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster

Trucks Deliverers

NiFi NiFi NiFi NiFi NiFi NiFi

On Delivery Routes

Courier service from the perspective of NiFi & MiNiFi

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster

Trucks Deliverers

Client Libraries

MiNiFi

MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi

Client Libraries

On Delivery Routes

Created to more effectively collect data at the edge

Apache NiFi

Data provenance

Designed for extension Visual command and control Flow templates Policy based security Clustering

Key Features

Data provenance

Different from Apache NiFi Design and Deploy Warm re-deploys

Key Features

Apache NiFi MiNiFi

Designed for extension

Command and Controlvs.

Design and Deploy

Different MiNiFi Distributions

Java– <40MB binary distribution– Requires Java 1.8– More feature complete– Targeted for any systems that can run a JVM (ie. Servers, Raspberry Pi)

C++– 600KB code size and static data ~50KB– Dynamic heap of ~1MB based on use-case– Targeted for resource constrained environments (ie. edge IoT devices)

Both use same config format and use NiFi terminology

Different focuses depending on requirements

Typical Simple ExampleTail a rolling file -> Site to Site

Simple Config.ymlTail a rolling file -> Site to Site

NiFi and MiNiFi, JAVA and C++

NiFi vs MiNiFi Java Processes

NiFi Framework

Components

MiNiFi

NiFi Framework

User Interface

Components

NiFi Java Processes

Bootstrap

bootstrap.conf

nifi.properties

flow.xml.gzreads &modifies

starts

NiFi MiNiFi

MiNiFi Java Processes

MiNiFi

Bootstrap

ConfigurationChange Notifier(s)

bootstrap.conf

nifi.properties

flow.xml.gzreads

starts

config.ymltransforms

NiFi MiNiFi

Same Extensible framework (nars)

NiFi standard processors are bundled– TailLog– UpdateAttribute– Route on content and attributes– PutEmail– ….

Allows MiNiFi Java to use NiFi processors

MiNiFi C++ Features Same design and deploy configuration Initial set of processors

– TailFile– GetFile– GenerateFlowFile– LogAttribute– ListenSyslog

Site to Site Client implementation in C++ for talking to NiFi instances

• Demo

• Next Step

Simple Config.ymlTail a rolling file -> Site to Site

Courier service from the perspective of NiFi & MiNiFi

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster

Trucks Deliverers

Client Libraries

MiNiFi

MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi

Client Libraries

On Delivery Routes

Demo: Digital Temp & Humidity Sensor, Raspberry Pi Zero, MiNiFi

Raspberry Pi Zero

with MiNiFi(link) USB hub for

power & WiFi

Digital Temp & Humidity Measurement Sensor

• Demo

• Next Step

MiNiFi Command and Control

Design Flow at a centralized place, deploy on the edge Version control of flows

– Align with NiFi SDLC work

Agent status monitoring Bi-directional command and control

Currently a feature proposal, initial version being architected

https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control

• Demo

• Next Step

CoreInfrastructure

Hortonworks DataFlow Use CasesRegional

InfrastructureSources

Dataflow Management• On-ramp into Hadoop• Log Collection / Splunk Optimization• Cyber Security• IoT Ingestion• Deliver data into stream processing engines

Real-time Event Processing (Kafka, Storm)

Move data between from on-prem and cloud environments

Apache MiNiFi Use Case: Distributed Systems: Retail POS

100’s or 1000’s or more terminals to gather: Real-time pricing info Real-time inventory updates Real-time offers Traceable logs of transactions Log of user-interactions for trouble-shooting

devices Predictive help desk

Apache MiNiFi Use Case: Distributed Systems: Connected Car

MiNiFi

SD Card

Raw data via CAN, Ethernet

Car Wireless Gateway

Buffer offline delay-tolerant data

Real-time data

Wi-Fi AP

DSRC RSU

eNodeB

Possibly modify route or adjust sampling rate and prepare to flush delay-tolerant buffers when SDRC RSU or Wi-Fi AP in range

Data Storage

Decision-Making Server

Map Database

Hortonworks maximizes intelligence with platforms that deliver actionable insights from connected car data

GENIVI Alliance: Open Source, In-Vehicle Infotainment Software

The GENIVI Alliance is a nonprofit industry alliance committed to driving the broad adoption of specified, open source, In-Vehicle Infotainment software.

Apache MiNiFi Use Cases: Distributed Systems of Devices

Routers Security cameras Cable modems ATM

Fleet of Trucks Manufacturing Line Security Appliance Point of Sale

Weather detection system Thermostats Utility/Power meters Fleet of Ships

Any distributed system of devices with data to be collected

Questions?

Learn more and join us!

Apache NiFi sitehttp://nifi.apache.org

Subproject MiNiFi sitehttp://nifi.apache.org/minifi/

Subscribe to and collaborate atdev@nifi.apache.orgusers@nifi.apache.org

Submit Ideas or Issueshttps://issues.apache.org/jira/browse/NIFIhttps://issues.apache.org/jira/browse/MINIFI

Follow us on Twitter@apachenifi

Questions?

Hortonworks Community Connection:Data Ingestion and Streaminghttps://community.hortonworks.com/

Thank you!

Hortonworks Data in Motion Webinar Series - Part 6, Edge Intelligence IoT MiniFi

Technology

© Hortonworks Inc. 2012 Hortonworks Page 1. © Hortonworks Inc. 2012 Big Data Changes the Game Megabytes Gigabytes Terabytes Petabytes Purchase detail

Hortonworks Data Platform - Workflow Management€¦ · Hortonworks Data Platform (December 10, 2019) Workflow Management docs.cloudera.com

Hortonworks apache training

Hortonworks roadshow

Hortonworks laurie maclachlan

Actian forrester- hortonworks

Future of Data Hortonworks Data Platform and Hortonworks ... · Hortonworks Connected Data Platforms and Solutions Hortonworks Connection Hortonworks Solutions Enterprise Data

Hortonworks Data Platform - Meetupfiles.meetup.com/13696062/HortonWorksOverview.pdf · Hortonworks Data Platform: Enterprise Hadoop The Hortonworks Data Platform (HDP) delivers all

Hortonworks Data Platform - Teradata Connector …...Hortonworks Data Platform November 30, 2018 2 2. Hortonworks Connector for Teradata 2.1. Introduction Hortonworks Connector for

Dataguise hortonworks insurance_feb25

Hortonworks DataFlow White Paper

NetApp Solutions for Hadoop - Hortonworks · NetApp Solutions for Hadoop Reference Architecture: Hortonworks . Faiz Abidi (NetApp), Ali Bajwa (Hortonworks), and Harsh Shah (Hortonworks)

MiNiFi 0.0.1 MeetUp talk

Hands-on with Apache NiFi and MiNiFi - Berlin Buzzwords · PDF fileHands-on with Apache NiFi and MiNiFi ... Content-Type: text/html Hello world! Standard FlowFileAttributes ... Precedent

Installing Hortonworks Sandbox Fusion on Machortonworks.com/wp-content/uploads/unversioned/pdfs/...Installing)Hortonworks)Sandbox)–)Fusion)on)a)Mac)1)! Installing Hortonworks Sandbox

Hortonworks hadoop big data_retail__white_paper

Hortonworks Cybersecurity Package - Installation

Introduction to Hortonworks Data Platform

Dell EMC Hortonworks Hadoop Solution · Dell EMC Hortonworks Hadoop Solution Node Architecture The Hortonworks Data Platform is composed of many Hadoop components covering a wide

HPE Reference Architecture for Hortonworks HDP 2.4 on HPE … · 2017-08-21 · Hortonworks Data Platform: A full Enterprise Hadoop Data Platform . Hortonworks Data Platform Version