Go With The DataflowApache NiFi

arien.vijn@linklight.nl

Telecom Architecture• Network Elements (NE) are managed by

Elements Management Systems (EMS)

• EMSes communicate with Network Management Systems (NMS)

• NMSes communicate with the OSS

• OSS communicates with the BSS

• Between systems are standardised interfaces, except for the South Bound Interface (SBI) of the EMS

Fulfilment

Assurance

Network Elements (NE / NEL)

Elements Management Systems (EMS / EML)

Network Management System(s)(NMS / NML)

Operational Support System (OSS / SML)

Business Support System (BSS / BML)

“The good thing about standards is that there are so many to choose from.”

― Andrew S. Tanenbaum

Reality• This architecture is not really implemented

Fulfilment

Assurance

NMS

Network Elements (NE)

EMS

BSS

OSS

Reality• This architecture is not really implemented

• Humans solve the missing parts and shortcomings

Fulfilment

Assurance

NMS

Network Elements (NE)

EMS

BSS

OSS

Reality• This architecture is not really implemented

• Humans solve the missing parts and shortcomings

• Often using Microsoft Excel

Fulfilment

Assurance

NMS

Network Elements (NE)

EMS

BSS

OSS

+

Reality• Only partly implemented

• Humans form the north and south bound interfaces over multiple layers

• Often using Microsoft Excel

• Or Python, PERL, awk, bash...

Fulfilment

Assurance

NMS

Network Elements (NE)

EMS

BSS

OSS

+

Reality• Only partly implemented

• Humans form the north and south bound interfaces over multiple layers

• Often using Microsoft Excel

• Or Python, PERL, awk, bash...

• Often in organisational silos

• Transport, delivery, IP, Consumer, B2B, etc.

Fulfilment

Assurance

NMS

Network Elements (NE)

EMS

BSS

OSS

Reality• Many systems

• Lots of data

• Little information

NMS

Network Elements (NE)

EMS

BSS

OSS

Data

? 2

dozensdozens

>300k

information

Transform Data Into Information

Transform Data Into Information• Dozens of systems

• Some open, some closed

• Tons of Network Elements

• NEs are off-limits due to change control

NMS

Network Elements (NE)

EMS

BSS

OSS

Data

information

? 2

dozensdozens

>300k

Transform Data Into Information• Dozens of systems

• Some open, some closed

• Tons of Network Elements

• NEs are off-limits due to change control

• Initial contract for 36 workdays NMS

Network Elements (NE)

EMS

BSS

OSS

Data

information

? 2

dozensdozens

>300k

36 workdays

Transform Data Into Information• R?

NMS

Network Elements (NE)

EMS

BSS

OSS

information

? 2

dozensdozens

>300k

36 workdays

NMS

Network Elements (NE)

EMS

BSS

OSS

information

? 2

dozensdozens

>300k

36 workdays

Transform Data Into Information• R?

• Python?

NMS

Network Elements (NE)

EMS

BSS

OSS

information

? 2

dozensdozens

>300k

36 workdays

Transform Data Into Information• R?

• Python?

• Go with the pros

NMS

Network Elements (NE)

EMS

BSS

OSS

information

? 2

dozensdozens

>300k

36 workdays

Transform Data Into Information• Python?

• R?

• Go with the pros

• Apache NiFi

• Based on NSA's Niagara Files project

• Flow-based programming

• and a little Python NMS

Network Elements (NE)

EMS

BSS

OSS

information

? 2

dozensdozens

>300k

36 workdays

+

Apache NiFi• Flow-based programming

• Bit like Lego Mindstorms

Apache NiFi• Flow-based programming

• Bit like Lego Mindstorms

• Drag processors on the canvas

• Put in a few settings

Apache NiFi• Flow-based programming

• Bit like Lego Mindstorms

• Drag processors on the canvas

• Put in a few settings

• Connect the processors

• Run...

• Add, change and work on it while the data flows.

Count The Number of Speakers

Count The Number of Speakers• We start with the GenerateFlowFile

processor

Count The Number of Speakers• We start with the GenerateFlowFile

processor

• We create an attribute called url and give it the value: https://nlnog.net/nlnog-day-2019/

Count The Number of Speakers• We start with the GenerateFlowFile

processor

• We create an attribute called url and give it the value: https://nlnog.net/nlnog-day-2019/

• Second processor gets the web page

Count The Number of Speakers• We start with the GenerateFlowFile

processor

• We create an attribute called url and give it the value: https://nlnog.net/nlnog-day-2019/

• Second processor gets the web page.

• Connect the processors

Count The Number of Speakers• Start the first processor

• One FlowFile in the queue between the processors

Count The Number of Speakers• Start the first processor

• One FlowFile in the queue between the processors

• We can view the contents and the history of the FlowFiles at any time

Count The Number of Speakers• Add the third processor and connect the

second

• Start the second processor

• Flowfile goes through

• Contains the HTML data of the NLNOG website

Count The Number of Speakers• Third processor: ExecuteStreamCommand

• Invokes a simple Python script that extracts the names of the speakers

• stdin to stdout

Count The Number of Speakers• Third processor: ExecuteStreamCommand

• Invokes a simple Python script that extracts the names of the speakers

• stdin to stdout

• Last processor:

• Counts the lines

Count The Number of Speakers• Third processor: ExecuteStreamCommand

• Invokes a simple Python script that extracts the names of the speakers

• stdin to stdout

• Last processor:

• Counts the lines

• 12!

Back to the Telecom Company• This is a very simple example

• NiFi is used for complex data extraction tasks

• Its graphical user interface helps with visualising the complexity

• It is used to combine data from many different systems

• We use it a lot to create time series data from systems that don't keep history

• Perhaps it is useful for you too

NMS

Network Elements (NE)

EMS

BSS

OSS

Information

? 2

dozensdozens

>300k

+

Thank you!