EMC REAL-TIME INTELLIGENCE (RTI) 3.1 - VMware · EMC REAL-TIME INTELLIGENCE (RTI) 3.1.0 Streaming Guide . April 2017 . Abstract This reference guide describes the RTI streaming platform

EMC REAL-TIME INTELLIGENCE (RTI) 3.1.0 Streaming Guide April 2017

Abstract

This reference guide describes the RTI streaming platform. It provides a step-by-step tutorial, reference material for the Streaming DSL and Component SDK, and a set of examples to use with the RTI-in-a-box Docker image.

302-003-752 REV 01

REFERENCE GUIDE

Copyright

The information in this publication is provided as is. Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable software license.

Copyright © April 2017 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. Other trademarks may be the property of their respective owners. Published in the USA April 2017 Reference Guide 302-003-752.

Dell Inc. believes the information in this document is accurate as of its publication date. The information is subject to change without notice.

2 EMC Real-Time Intelligence (RTI) 3.1.0 Streaming Guide

Contents

Contents

Chapter 1 Streaming Overview 5 Introducing RTI Streams ....................................................................................... 6 RTI Streams concepts ........................................................................................... 7 Key architecture features ...................................................................................... 9

Chapter 2 Deploying RTI Stream Computations 12 Overview ............................................................................................................. 13 Starting the RTI virtualized environment .............................................................. 13 Before running the examples .............................................................................. 16 Running an example ........................................................................................... 18 Command-line interface ...................................................................................... 23

Chapter 3 Reference 25 Stream structure ................................................................................................. 26 Core attributes .................................................................................................... 27 Event processing ................................................................................................ 29 Data distribution endpoints .................................................................................. 29 Formatter DSL .................................................................................................... 39 Component SDK ................................................................................................. 40 Component libraries ............................................................................................ 54 Telecom components .......................................................................................... 71 Stream management ........................................................................................... 85

Chapter 4 Examples 95 Event anonymization ........................................................................................... 96 Dynamic code deployment .................................................................................. 98 Geofence detection ........................................................................................... 100 Location subscriber opt-in streams .................................................................... 104 Spring stream flow ............................................................................................ 109 Spring Integration endpoint ............................................................................... 111 Multiple endpoints ............................................................................................. 115 Event enrichment .............................................................................................. 119 Telco event enrichment ..................................................................................... 123 Number attribute enrichment ............................................................................. 134 Condition enrichment ........................................................................................ 137 Component SDK ............................................................................................... 139


Contents EMC Confidential [delete if not required]

Kafka output ...................................................................................................... 145 Event Stream Subscription (ESS) ..................................................................... 147 Non-telco streams ............................................................................................. 151 KPI stores ......................................................................................................... 152


Chapter 1: Streaming Overview

Chapter 1 Streaming Overview

This chapter presents the following topics:

Introducing RTI Streams ..................................................................................... 6

RTI Streams concepts ........................................................................................ 7

Key architecture features ................................................................................... 9



Introducing RTI Streams RTI Streams is a high performance, declarative, distributed stream processing framework, as shown in Figure 1. Stream processing is a computing abstraction that enables processing large volumes of data as it is generated in real time. This data can be user activity updates (webpage views, impression / click streams from display ads), exchange feeds, or database logs.

Figure 1. RTI stream processing platform

RTI Streams makes stream processing attainable for all developers, regardless of programming experience and business knowledge. RTI Streams is an all-in-one solution that enables you to spend your time working on business domain solutions instead of managing a complex infrastructure.

RTI is an event-driven, reactive programming platform. Developers build component modules that are linked together as a series of operations, to form a processing stream. Each stream is connected to other streams, by globally unique stream names, to create a streaming topology.

The framework connects streams similarly to a publish–subscribe architecture. Each stream defines a list of dependent streams to subscribe to and the framework ensures that each stream message that is produced is directed to the subscribing stream.

RTI Streams enables you to update streams and deploy new streams independently of one another, even while the streams are running in a production environment. Unlike other streaming frameworks that require you to define the topology statically in your program’s main method, RTI Streams enables you to be flexible with a stream topology. You can submit streams to a cluster, by using the RTISH command line interface tool. For example:

rtish> rti stream deploy --definition customStream.json

Event-driven architecture

Real-time stream deployment



All active streams run in every RTI cluster member as a distributed process. This design guarantees that a stream performs data processing locally in each process, and provides high availability if the process fails.

The RTI platform is built on a high-performance data grid. Because data locality is a key factor in performance, events are splayed across the cluster using data affinity rules. This design ensures that streams operate on their set of events without expensive out-of-process I/O calls.

Non-blocking publish-subscribe channels in the same JVM provide stream-to-stream communication and the low latency characteristics of a high-performance computing (HPC) platform. Processing in a stream is based on component chaining and, therefore, is in the same process. Reference data is colocated with the stream to enable fast look-ups and event enrichment.

Because RTI is highly available, your stream continues to run even if there is a failure. If a cluster node fails, another cluster member starts to process the events by using a load balancing strategy.

Guarantees of RTI messages are a best effort, at least once. If there is an unrecoverable failure, the system does not process any messages missed during the outage.

RTI Streams concepts RTI Streams is an RTI architecture sub-system. The architecture enables you to deploy and manage multiple streams in every RTI instance as a distributed process. That is, RTI Streams is a dynamic distributed process architecture.

The core components that make up the Streams framework include:

• Stream DSL—defines and encapsulates the processing requirements as a stream process. Elements such as event source subscriptions, processing logic, and distribution endpoints are definable using the Declarative Streaming Language (DSL).

• Component API—enables you to build and extend the shipped component library for your own domain solution requirements.

• ProtocolEvent object—is the core event that is passed and operated on in a component.

• EventView object—provides a distinct set of attributes, or view, which is sent to distribution endpoints.

When you build streams to deploy to the RTI system, use the component DSLs that are provided or extend the library with new components to implement business use cases.

The Stream DSL enables you to define domain-specific low latency event processing streams to deploy at run time across the distributed RTI platform. A stream consists of event subscriptions, flow processing components, and distribution endpoints. To manage streams across the grid, use RTISH or the Streams REST API.

Performance

Fault tolerance

Stream DSL



The key stream DSL elements include:

• name—The globally unique identifier of the stream.

• using—(Event subscription) The list of event sources either from pre-defined event sources or deployed streams.

• flow—(Processing logic) A sequence of components that act on the ProtocolEvent object.

• endpoint—(Data distribution) A list of distribution endpoints to send the computed result on to external systems.

The following example shows key stream DSL elements:

For more information about the streaming DSL, refer to Stream structure.

The Component API provides a suite of developer APIs that enables you to extend the client component libraries for specific domain requirements. For more information about how to build custom components, refer to Component SDK.

The key event object used in the streaming platform is the ProtocolEvent. For information about how to access and modify the event object, refer to Chapter 3.

{ "name" : "vipCustomerAnonymizedStream", "using" : ["defaultFireHose"], "flow" : [ { "expressionFilter": { "expression": "event.getProtocolDetails().get('vipcustomer') == true && event.getProtocolDetails().get('zipcode') = '02201'", "lang" : "GROOVY" } }, { "anonymization": { "algorithm" : "md5", "fields": ["surname","accountNumber", "address"] } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "amqp": { "formatter": { "type": "JSON" }, "exchange": "CUSTOMERS", "routingKey": "VIP.BOSTON" } } ] } }

Component API

ProtocolEvent record



An EventViewObject is a view object that represents a projection of an attribute selection expression. If you define a list of attributes, this object is used to create a view, otherwise a ProtocolEvent is used. This object is sent to the distribution endpoint formatters for further processing.

Key architecture features Key architecture features of RTI include:

• Declarative stream definition DSL

• Scalable stream processing

• Concurrent stream processing

• Out-of-Order event processing in a stream

• Event isolation using deep copy cloning

• Stream dependency management

• Dynamic runtime stream construction

• Low latency processing that is based on data colocation and event distribution affinity rules

• Stream isolation in the event of failures

• Stream performance metrics

• Restartable streams on system restart and on new cluster member additions

• Dynamic scripting compilation for supported languages

The streaming platform is a distributed platform framework, as shown in Figure 2. A stream is deployed to every service grid member where the stream registry ensures event subscription dependencies are valid and registers the stream to the cluster member process.

Event view object

Stream platform



Figure 2. Distributed platform framework

The stream registry ensures that streams are valid before being deployed, for example, to ensure that the stream dependencies exist. The stream registry also acts as a global registry of all deployed streams. The registry is replicated to each service grid member for high availability, fault tolerance, and to enable local registry processing for fast start-up timing. If new members are added to the service cluster at runtime, the registry is replicated so that streams can be auto deployed.

The administration service performs the management aspects of the streaming platform such as deploy, enable, disable, undeploy, list and describe.

The Streaming registry manages dependencies across streams to enable construction of complex topologies. This management of dependencies allows streams and metrics to be dependent on each other to promote computed event reuse and complex topologies. Each stream topology is executed independently in each cluster member to provide high throughput event processing. Source events are initially routed by using an event routing algorithm that is based on the ProtocolEvent primary key.

Stream registry

Administration service

Stream topology



Figure 3 shows that the stream graph is distributed to every cluster member.

Figure 3. Stream graph distribution

Member stream topology


Chapter 2: Deploying RTI Stream Computations

Chapter 2 Deploying RTI Stream Computations


Overview ............................................................................................................ 13

Starting the RTI virtualized environment ......................................................... 13

Before running the examples ........................................................................... 16

Running an example ......................................................................................... 18

Command-line interface ................................................................................... 23



Overview The RTI installation includes a virtual environment (Docker image) and other required components that are installed and configured. The image includes RTI Streaming API examples.

The Docker Machine must be installed on your system. Refer to the Docker website for information about Docker administration.

The RTI Docker image comes pre-installed and configured with the following:

• CentOS operating system

• OpenSSH

• Oracle Java JDK 1.7

• System tools such as Netcat

• RabbitMQ software

• RTI Core binaries

• Additional RTI libraries for custom features

• RTI Streaming API examples

• Kafka and Zookeeper

• Scripts to operate RTI, RabbitMQ, and Kafka

The following steps and examples provide a tutorial for deploying an RTI Stream computation.

Starting the RTI virtualized environment 1. Download RTI-in-a-box

Contact your RTI representative for the RTI image, distributed as a compressed tar file named rti-version-docker.tar.gz where version is the RTI core version number.

2. Decompress the image file and place it on the same system that has the Docker Machine installed and running. The following example shows how to use gzip to extract the decompressed rti-version-docker.tar file.

$ gzip –d rti-version-docker.tar.gz

3. Load the image from rti-version-docker.tar by running the following command:

$ docker load –i rti-version-docker.tar

4. Start the container by running the following command, which also maps exposed ports:



$ docker run -d -p 2022:22 -p 5672:5672 -p 7080:7080 -p 15672:15672 -p 10071:10071 -p 10072:10072 -p 10073:10073 -p 10074:10074 -p 10075:10075 -t --name rti-version-docker rti-version-docker

This step creates a container named rti-version-docker. Verify the details by running:

$ docker ps –a

5. Log in to the container by running the following command, using rti/emc as the user/password login credentials:

$ ssh rti@your_docker_machine_ip –p 2022

Running the docker-machine ip command on your Docker Machine host returns the IP address.

If this is the first time that you access Secure Shell (SSH), type yes when you are queried about continuing to connect.

At login, you are in the /opt directory, which includes the following:

RTI is installed in the emc_rti_300/rti-t directory, which this document refers to as RTI_HOME. All scripts required to operate RTI (such as start, stop, and the like) are in the emc_rti_dist/run_rti directory. Scripts to manage Kafka are in the emc_rti_dist/run_kafka directory. RabbitMQ is running as a service when you start the container.

Other scripts that are included with the installation are not required to run the examples in this document.

6. Connect to RTI processes using a JMX client such as VisualVM or JConsole:

a. Obtain the Docker IP address by running the following command on the system on which you have the Docker Machine installed:

$ docker-machine ip

b. If the IP address is 192.168.99.100, go to step 7. Otherwise, go to the following substep.

c. Modify the /opt/emc_rti_dist/run_rti/rtish_base_apps.cmd script to set the value of the Java command line option for the RMI server to the IP you obtained. Search for the -Djava.rmi.server.hostname string and replace the default value (192.168.99.100) with your system’s IP.

Repeat this step for all rti set env lines that have this property. Each line sets the options for an RTI process. As described in a subsequent section, you will connect your JMX client to this IP address.

[rti@your_host opt]$ ls container-startup.log emc_rti_300 init-rabbitmq.sh kafka_2.11-0.9.0.1 docker-entrypoint.sh emc_rti_dist init-rti.sh



7. Run the following script file to start all the RTI components, launch the RTI processes, and load the reference data that is required for the Streaming examples:

[rti@your_host opt]$ emc_rti_dist/run_rti/rti-start.sh

8. Start the RTISH command line interface (CLI) tool using the following script and list RTI processes to verify that RTI is running

Note: RTISH is in the $RTI_HOME/common/rtish/bin directory. You can run the following script, which invokes the same RTISH binary.

Your system displays output similar to the following example:

[rti@your_host opt]$ emc_rti_dist/run_rti/rtish.sh rti starting 'rti.shell' on your_host... ___ ________________ __ / _ \/_ __/ _/ __/ // / / , _/ / / _/ /_\ \/ _ / /_/|_| /_/ /___/___/_//_/ v3.1.0 rtish> rti list processes Local processes: 72090 io.pivotal.rti.boot.RTI ingestgrid.app start 71438 io.pivotal.rti.boot.RTI distgrid.app start 71037 io.pivotal.rti.boot.RTI locator.app start 70817 io.pivotal.rti.boot.RTI locator.app start 72586 io.pivotal.rti.boot.RTI provisioning.app start 72592 org.springframework.boot.loader.PropertiesLaunchar restapi.app start 23115 io.pivotal.rti.boot.RTI rti.shell

Note: For RTI release 3.1.0 and later, the ingest grid acts as an ingester by default. This functionality reduces communications overhead and improves performance in the cluster. Because the RTI Docker image is configured to use the ingest grid as the ingester, there is no separate ingester process. EMC recommends that you run in this mode only. For information about how to use a separate ingester application, refer to the EMC Real-Time Intelligence (RTI) Administrators Guide. You must modify the scripts listed in the rti-start.sh file.

9. Run the following script to start Kafka:

[rti@your_host opt]$ emc_rti_dist/run_kafka/kafka-start.sh

This is only needed for one of the Streaming API examples that outputs to Kafka.

10. Run the Streaming API examples, which are in the $RTI_HOME/examples/streaming-api/ directory.

Each example is in a separate subdirectory. Sample data and reference data files for all the examples are provided in sampleData and the referenceData directories respectively, both under streaming-api. Stream the sample data through the platform. Reference data is loaded when you run RTI, as shown in step 8.



The next section describes the steps to run the examples. For a detailed description of each example and the Streaming API capability it demonstrates, refer to Chapter 4. The examples refer to several out-of-the-box (OOTB) components to enable streams to be built and deployed. For more information, refer to Component Libraries.

Before running the examples Refer to Chapter 4 for a detailed description about each example and the Streaming API capability it demonstrates. The examples refer to several OOTB components that enable you to build and deploy streams. Refer to Component libraries for more information.

The Docker image has been configured to expose ports for JMX monitoring of RTI processes. The following table lists the RTI processes and corresponding JMX ports.

Table 1. RTI processes and corresponding JMX ports

RTI process JMX Port

Ingest Grid 10072

Service(Distribution) Grid 10073

Provisioning Manager 10074

REST API 10075

Port 10071 is also exposed. If the ingester is run as a separate process, it can use this port.

Using a JMX client such as VisualVM or JConsole, connect to the container.

Note: These steps are general steps. For more information, refer to the documentation for your JMX client.

1. Connect to the RTI Docker containers using the IP address that you obtained earlier (refer to Connect to RTI processes using a JMX client such as VisualVM or JConsole:. For example, in VisualVM, connect to a “Remote Host”.

2. Add JMX connections to the remote host ports that are listed in Table 1. When connected, the RTI processes are displayed.

3. Use the client’s MBeans browser to view details for each process.

JMX monitoring


http://dcade.lss.emc.com/confluence/display/CTORTI/Component+Libraries


Figure 4 shows monitoring with VisualVM.

Figure 4. Monitoring with VisualVM

For information about Streaming API-related JMX monitoring, refer to

Stream metrics and Monitoring streams. For general information about RTI JMX monitoring, refer to the EMC Real-Time Intelligence (RTI) Administration Guide.

To stop RTI:

1. Go to the /opt/emc_rti_dist/run_rti directory.

2. Run the rti-stop.sh script, which stops all running RTI processes.

3. To start RTI again, run the rti-start.sh script. The following details about the host and some initialization scripts are useful to note. Because the scripts ran when the image was created, it is not necessary to run the scripts to run the examples.

• The Root user/password login credentials for the container are root/centos.

• The /opt directory includes the init-rti.sh script that installs RTI.

• The /opt/emc_rti_dist/run_rti directory includes an rti-clean.sh script that uninstalls RTI. Only run this script if absolutely necessary.

• The /opt directory includes the init-rabbitmq.sh script that creates the RabbitMQ user and assigns privileges. The /opt directory includes the docker-entrypoint.sh script that sets the Docker container entry point. There is no need to run these scripts.

Stopping RTI

Initialization scripts



Running an example To run an example, follow these steps:

1. Connect to the service grid (first time only)

2. Deploy the stream

3. Pass sample data

4. Verify outputs

Our first example stream topology demonstrates the flow of events through the RTI system from the ingest grid to the file and Advanced Message Queuing Protocol (AMQP outputs.

For the first time only, connect to the service grid.

1. Change to the $RTI_HOME directory:

[rti@your_host opt]$ cd $RTI_HOME

All the example files are referenced relative to this location.

2. Start the RTISH tool and connect to the service grid as shown in the following example:

rti@your_host rti-t]$ /opt/emc_rti_dist/run_rti/rtish.sh rti starting 'rti.shell' on your_host... ___ ________________ __ / _ \/_ __/ _/ __/ // / / , _/ / / _/ /_\ \/ _ / /_/|_| /_/ /___/___/_//_/ v3.1.0 rtish> rti refdata connect --host localhost --port 10335 Connected to: localhost[10335].

When you have established a connection to the service grid, deploy a stream in to the platform.

The following example shows the $RTI_HOME/examples/streaming-api/allData/allDataStream.json file, which is the stream specification:

Connect to the service grid

Deploy a stream



{ "name" : "allDataStream", "using" : ["defaultFirehose"], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/allData" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }



1. Deploy the stream as follows:

2. List the streams to ensure that they are active:

3. View details about the processing and the target endpoints to which the events are sent.

The example shows that the stream subscribes to events from the default firehose, and passes all events without modification to the outputs in CSV format. The outputs are configured as a file as well as an AMQP exchange. Batches of events are written to a CSV file created at {rti.data}/allData on the service grid instance. {rti.data}

rtish> rti stream deploy --definition examples/streaming-api/allData/allDataStream.json Deployed allDataStream to system. Starting up for processing. allDataStream successfully started.

rtish> rti stream list Name Active -------------------------------------------------------------- allDataStream true

rtish> rti stream describe --name allDataStream Attribute Value ------------------------------------------------------------------------------------------- name allDataStream enabled true using [defaultFirehose] cloneEvent false queueSize 1000 workerThreads 1 Endpoint - select * selectType CASCADE_HASH type FILE - Formatter - type csv encoding UTF-8 batchSize 8192 batchTimeout 15000 fileDateFormat yyyyMMdd'T'HHmmssSSS fileBasePath ${rti.data}/allData compress false compressionDelay 15000 type AMQP - Formatter - type csv encoding UTF-8 exchange DECODER.STREAM.CUSTOM routingKey ALL batchSize 250 batchTimeout 100 Setting - username guest password guest hostAddresses [localhost] virtualHost / connectionTimeoutMillis 10000 heartbeatSeconds 10 channelCacheSize 50



resolves to $RTI_HOME/var/data/service_grid_instance_name. The same events also appear on the AMQP exchange named DECODER.STREAM.CUSTOM using the ALL routing key.

To create a queue:

1. Use your browser and enter http://your_docker_machine_ip:15672.

2. Log in using rti/emc as the RabbitMQ user/password login credentials.

3. Click on the Exchanges tab, and then select the DECODER.STREAM.CUSTOM exchange.

This exchange is automatically created when the stream is deployed.

4. Create a queue and bind it to the DECODER.STREAM.CUSTOM exchange using the ALL routing key.

You can view output events flowing to this queue.

For information about how to create queues, refer to the RabbitMQ documentation.

The $RTI_HOME/examples/streaming-api/sampleData directory includes sample events for all the examples.

1. Open a new terminal window and log in (ssh) to the RTI container. For information about logging in to the RTI container, refer to Starting the RTI virtualized environment.

2. Go to the sampleData directory:

[rti@dc889d506a42 opt]$ cd $RTI_HOME/examples/streaming-api/sampleData

3. Run the following event in sampleData.json:

To stream the event through RTI, run the following command:

The Netcat utility is provided as part of the RTI container and sample events in JSON format are sent to the JSON adapter port.

The event flows through RTI and the deployed allDataStream. It is written to a CSV file in the $RTI_HOME/var/data/distgrid-one/allData directory.

1. Open a new terminal window and log in (ssh) to the RTI container. For information about logging in to the RTI container, refer to Starting the RTI virtualized environment.

2. Change to the allData directory.

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"359254061234567","eventStartTime":1425556331,"protocolDetails":{"mccmnc":"23415","lac":"178","cellId":"20108"}}

[rti@your_host sampleData]$ nc -v 127.0.0.1 29009 < sampleData.json Ncat: Version 6.40 ( http://nmap.org/ncat ) Ncat: Connected to 127.0.0.1:29009. Ncat: 203 bytes sent, 0 bytes received in 0.20 seconds.

Create a RabbitMQ queue

Start the event feed

Check the file output



The service grid instance is distgrid-one and allData is the folder specified in the stream specification for the output events. Each example stream has a separate output folder that is created under $RTI_HOME/var/data/distgrid-one when the stream is deployed.

The following example shows the output event. The same event flows through without any modification.

Select the queue that you created that is bound to the DECODER.STREAM.CUSTOM exchange with the ALL routing key. Click Get Message(s) to view the same event as shown in the following example.

[rti@your_host opt]$ cd $RTI_HOME/var/data/distgrid-one [rti@your_host distgrid-one]$ ls allData cacheserver diskstore gateway lands lors rti.pid [rti@ your_host distgrid-one]$ cd allData/ [rti@ your_host allData]$ ls allDataStream-distgrid-one@your_host.204-20161107T145447150.csv

bb1a83d0-a778-11e6-0d72-0266787ed067,234158013224667,447795470718,,359254061234567,1478804765832,1425556331,0,0_0,178 _20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realImsi =;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[]

Check the RabbitMQ dashboard



Command-line interface RTI provides RTISH, a common CLI to manage RTI. The Streams platform has a dedicated command set to manage user streams. For more information about the stream CLI functions, refer to Stream management.

Deploying a stream in to the RTI distributed system requires a stream.json definition file and connection to the services cluster using RTISH.

Undeploying a stream requires a valid stream name and connection to the services cluster.

JMX metrics are enabled for each deployed stream. These metrics appear in their own JMX branch using the io.pivotal.rti.streams namespace. Use standard JMX visualization tools such as JConsole or VisualVM to view Stream metrics by connecting to a service grid member.

Typical metrics include:

• Number of events processed

• Number of events discarded

• Number of events distributed

• Component level metrics.

# Deploy a distributed stream rtish> rti stream deploy --definition customStream.json Deployed customStream to system. Starting up for processing. customStream successfully started.

# Undeploy a distributed stream rtish> rti stream undeploy --name customStream customStream successfully undeployed.

Deploy a stream

Undeploy a stream

Stream metrics



Figure 5 shows the allDataStream from Running an example being monitored by using VisualVM.

Figure 5. allDataStream example

For more information about monitoring, refer to Monitoring streams.


Chapter 3: Reference

Chapter 3 Reference


Core attributes .................................................................................................. 27

Event processing .............................................................................................. 29

Data distribution endpoints .............................................................................. 29

Formatter DSL ................................................................................................... 39

Component SDK................................................................................................ 40

Component libraries ......................................................................................... 54

Telecom components ....................................................................................... 71

Stream management ......................................................................................... 85



Stream structure The Streams DSL consists of three sections:

• Event subscription and core attributes

The subscription (core attributes) section enables a set of functional and non-functional characteristics for a stream. Setting such event subscriptions, queue depth, event cloning and concurrency control attributes provide the necessary features to work with real time data.

• Processing

The processing (flow) section enables the developer to define a sequence of computational processes to act on an event. A set of one or more components are defined in the flow, using component DSL syntax. Components can be OOTB or custom built for domain specific computations.

• Distribution

The distribution (endpoint) section provides the developer to define distribution endpoints for computed events. Functions such as event attribute projections and formatting are supported. Multiple endpoints can be defined to enable various event projections to be sent to selected output channels.

The flow and distribution sections are optional although at least one of these sections needs to be present for a valid stream configuration.

Figure 6 shows the general event flow for a stream.

Figure 6. Stream event flow

The following example subscribes to the defaultFirehose event sink and filters events based on a value greater than zero before passing the event as a CSV formatted object to the AMQP exchange.

General stream definition



Core attributes Core attributes enable a set of functional and non-functional characteristics for a stream. Setting event subscriptions and queue depth provide the necessary features to work with real-time data.

name attribute (Required) All streams require a unique name that will be used throughout the system. This name is used to execute stream management functions such as deploy, undeploy, enable, disable, and describe. Another function of the name is to provide the ability to chain streams together to increase reusability. Refer to using attribute.

"name" : "unique_stream_name"

enabled attribute (Optional) Enables or disables a stream from processing events. All dependent streams are affected. The default is enabled (true).

"enabled" : true

{ "name" : "customStream", "using" : ["defaultFirehose"], "flow" : [ { "filter": { "expression": "protocolDetails['value'] > 0" } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "amqp": { "formatter": { "type": "CSV" }, "exchange": "COMPANY.BUSINESS.LINE", "routingKey": "CUSTOM.USER_DEFINED" } } ] } }



using attribute (Optional) Provides the event subscription mechanism to enable events to be processed from a dependent event source, stream chaining. The attribute takes a list of stream names or system event sources. System event sources include:

• defaultFirehose—provides a ProtocolEvent object as the payload. This value is the default. If a stream is designed to be subscribed by metrics streams, and it uses defaultFirehose as the event source, include a ProtocolEventToSubscriber component at the end of the flow definition. For more information, refer to Transformers.

• subscriberStream—provides a Subscriber object as the payload and is used for Telecom stream output only. EMC recommends that you include only the endpoint component in a stream that uses subscriberStream as the event source.

The following example subscribes to events from two sources: the default firehose defined by RTI (all events) and the output of an existing stream named some_other_stream_name.

"using" : ["defaultFirehose","some_other_stream_name"]

cloneEvent attribute (Optional) When enabled, clones an event before flow processing occurs. By default, this attribute is disabled.

A clone is a deep copy of all event attributes. Enabling this attribute guarantees that all modifications to the event are only seen by the owning stream therefore providing event modification isolation. If disabled, any modifications on the event by a stream would be seen by all other processing streams with the same setting.

"cloneEvent" : true

Note: Cloning might require additional memory for the JVM. Review event rates and the JVM memory configuration setting to ensure that the platform is sized correctly.

queueSize attribute (Optional) Defines the number of events to buffer for the processing flow. When the queue is full, any events to be placed in it are blocked until a free slot is present. Consider changing this value when running multiple streams or if a stream is computationally intensive. The default queue size is set to 1000 events.

"queueSize" : 5000

workers attribute (Optional) Defines the number of workers that process data from the queue. If the value is one, which is the default value, processing is synchronous. If the value is greater than one, multiple events are processed concurrently. Change this behavior when event processing ordering is not required.

"workers" : 100



Note: Increasing worker threads incurs greater CPU utilization.

eventAccessProtection attribute (Optional) If enabled, places the event under protected access control to ensure multiple streams can interact with the same underlying event. By default, this attribute is disabled (false). Any stream process that uses the same event and attribute setting can modify an event attribute.

"eventAccessProtection" : true

Event processing Implement real-time event flows by using the Declarative Stream Language (DSL). The flow section is optional if an endpoint has been defined.

A flow encapsulates a set of reusable components that are defined in a logical sequence that can execute using real-time event data. The following example filters events that are based on type, and then anonymizes sensitive fields before further downstream processing.

RTI provides a library of reusable components that can be used to construct a processing flow. Additionally, RTI provides a Component API for developers to build custom components. For more information about the API, refer to Component API . For more information about available OOTB components, refer to Component libraries.

Data distribution endpoints An endpoint defines an event projection and the output transports, including formatting. As a core feature, a stream can have zero or more endpoint definitions to enable a single event to be published to independent output transports with a unique event select projection.

Current supported output transports include AMQP, Kafka, file, and Spring Integration. Spring Integration provides the extensibility to publish to a wide range of target event sinks such as MongoDB, Neo4J, GreenPlum, and the like.

"flow" : [ { "protocolFilter": { "type" : "vipCustomer" } }, { "anoymisation": { "algorithm" : "md5", "fields": ["surname","firstname", "address", "accountcode"] } } ]

What is a flow?

Components



The endpoint structure consists of two sections: selector and outputs. The following example shows the general structure for an endpoint.

"endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "amqp": { "formatter": { "type": "JSON" }, amqp properties ......... }, }, { "file": { "formatter": { "type": "CSV" }, file properties ......... } } ] }

(Required) Defines a list of event attributes to be sent to the defined outputs. The selector determines what attributes to pass on to the output formatter. A selector can be considered a type of projection similar to a SQL select statement.

The selector expression can be a comma separated list of event attributes or the standard select all wildcard (*) for all event attributes. When you specify a comma-separated list, a cascading search on the ProtocolEvent object looks up attributes.

All event attributes To output all attributes in a ProtocolEvent, use the following expression:

Select a set of attributes To output a selected set of attributes from a ProtocolEvent object, use the following expression:

Note: If the selector expression is set to all attributes, the configured outputs specification receives a ProtocolEvent. Otherwise, an EventViewObject containing the set of selected attributes is received. This is important when building custom formatters. For more information, refer to Formatter DSL.

"selector" : { "expression" : "*" }

"selector" : { "expression" : "firstname, surname, address, city" }

Endpoint structure

Selector section



(Required) Defines a list of outputs to which an event will be sent. For example:

Events are processed concurrently over the list of defined output channels.

The following shows an example in which all event fields are output to AMQP in JSON format and to a CSV file, using various batch sizes and timeouts.

Note: If you defined a flow, the endpoint is optional. Otherwise, you must define an endpoint.

"outputs" : [ { "amqp": { "formatter": { "type": "JSON" }, "exchange": "STREAMS.CUSTOM", "routingKey": "A_ROUTING_KEY", "batchSize": 250, "batchTimeout": 100 } } ]

"endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "amqp": { "formatter": { "type": "JSON" }, "exchange": "STREAMS.CUSTOM", "routingKey": "A_ROUTING_KEY", "batchSize": 250, "batchTimeout": 100 } }, { "file": { "formatter": { "type": "CSV" }, "fileBasePath" : "/home/rti/data", "compress" : true, "compressionDelay" : 15000, "batchSize" : 10000, "batchTimeout" : 15000 } }, { "spring" : { "formatter": { "type": "CSV" }, "file": "./springIntegrationEndpoint.xml" } } ] } }

Outputs



This output specification enables you to distribute resulting events using AMQP. You can define the various AMQP attributes at run time.

The following example shows the output specification:

Attributes

The following lists the attributes used for this endpoint.

formatter attribute (Optional) Sets the type of formatting to use. The default value is CSV formatting. For more information, refer to Formatter DSL.

exchange attribute (Required) Sets the exchange to which the event is sent.

"exchange": "EXCHANGE.STREAM.CUSTOM"

routingKey attribute (Optional) Sets the event routing key that determines how messages are routed to flow to a queue in RabbitMQ.

"routingKey": "EVENT_ROUTING_KEY"

batchSize attribute (Optional) Sets the number of events to batch in a single AMQP message. The default value is 8192 batched events.

"batchSize": 250

batchTimeout attribute (Optional) Sets the timeout to trigger a batched AMQP event to be sent. If the batch size is exceeded first, then the batched events are sent. Otherwise, the events are sent when the timeout expire. The setting is in milliseconds and the default value is 4000 ms (4 seconds).

"amqp": { "formatter": { "type": "CSV" }, "exchange": "EXCHANGE.STREAM.CUSTOM", "routingKey": "EVENT_ROUTING_KEY", "batchSize": 250, "batchTimeout": 100, "deliveryModePersistent" : true, "connectionSetting" : { "username" : "test", "password" : "test", "hostAddress" : "localhost:5672", "virtualHost" : "/", "channelCacheSize" : 50, "heartbeatSeconds" : 10, "connectionTimeoutMillis" : 30000 } }

AMQP



"batchTimeout": 100

deliveryModePersistent attribute (Optional) Sets disk persistence for the AMQP queue. By default, this attribute is disabled (false).

"deliveryModePersistent" : true

Connection Setting Section (Optional) Enables you to define the AMQP host for RTI event messages. If this section is not specified the global system AMQP settings will be applied.

username (Optional) Sets the username. The default account is guest.

"username" : "test"

password (Optional) Sets the password. The default password is guest.

"password" : "test"

hostAddress (Optional) Sets the hostname to which events are sent. The format is hostname:port. The default host is localhost:5672.

"hostAddress" : "localhost:5672"

hostAddresses (Optional) Sets the multiple hosts to which events are sent. This attribute overrides the hostAddress attribute. The default host is localhost:5672.

"hostAddresses" : [ "localhost:5672", "drhost:5672" ]

virtualHost (Optional) Sets the virtual host to use. The default is /.

"virtualHost" : "/"

"amqp": { "connectionSetting" : { "username" : "test", "password" : "test", "hostAddress" : "localhost:5672", "virtualHost" : "/", "channelCacheSize" : 50, "heartbeatSeconds" : 10, "connectionTimeoutMillis" : 30000 } }



channelCacheSize (Optional) Sets the maximum currently configured channels that can be cached in the idle state. The default value is set to 25.

"channelCacheSize" : 50

heartbeatSeconds (Optional) Sets the requested heartbeat interval on the underlying RabbitMQ connection. The default value set to a 10 second interval.

"heartbeatSeconds" : 10

connectionTimeoutMillis (Optional) Sets connection establishment timeout in milliseconds. Supported values are zero and greater.

"connectionTimeoutMillis" : 30000

This output specification enables you to distribute resulting events by using Kafka. You can define the various Kafka attributes at run time.


Attributes The following lists the attributes used for this endpoint.

formatter attribute (Optional) Sets the type of formatting to use. The default value is CSV formatting. For more information, refer to Formatter DSL.

"kafka": { "formatter": { "type": "CSV" }, "bootstrapServers": ["localhost:9092"], "clientId": "demoRTI", "compressionType": "gzip", "batchSize": 8192, "bufferMemory": 16777216, "acks": "all", "topic": "TOPIC.STREAM.CUSTOM", "partition": 1, "fixedKey": "myOU", "propertyKey": "imsi", "createTopic": true, "zookeeperServers": ["localhost:2181"] "zookeeperSessionTimeout": 7500, "zookeeperConnectionTimeout": 5000, "topicPartitions": 2, "topicReplicationFactor": 2 }

Kafka



bootstrapServers attribute (Required) Sets the list of Kafka servers to which events are sent. The format is a list of servers. Each server in the list is in the form of hostname:port.

"bootstrapServers": ["localhost:9092"]

clientId attribute (Optional) Sets the client ID that the Kafka producer passes to the Kafka server. The client ID helps track the request source in Kafka logs. If you do not assign an ID, Kafka automatically assigns a unique client ID.

"clientId": "demoRTI"

compressionType attribute (Optional) Sets the compression type to use for communication with the Kafka server. Supported values include none, gzip, lz4, or snappy. The default value is none.

"compressionType": "gzip"

batchSize attribute (Optional) Sets the maximum number of events to batch before sending to the Kafka server. The producer might send fewer messages in a batch. The default value is 16,394 batched events.

"batchSize": 8192

bufferMemory attribute

(Optional) Sets the total bytes of memory the component can use to buffer events before sending to the Kafka server. The default value is 33,554,432 bytes (32 Megabytes).

"bufferMemory": 16777216

acks attribute (Optional) Sets the number of acknowledgements that the Kafka leader must receive before considering the request to be complete. If the value is set to:

• 0, the component does not wait for any acknowledgments from the server.

• 1, the component waits for the leader to write the record to its local log only. This is the default.

• all, the component waits for the leader and all its followers to write the record to their logs, guaranteeing the record will not be lost

"acks": "all"

topic attribute (Required) Sets the Kafka topic to which the events will be sent.

"topic": "TOPIC.STREAM.CUSTOM"



partition attribute

(Optional) Sets the Kafka partition to which the events will be sent. This value must be a non-negative integer. Kafka selects the partition based on the key (if one is specified). If neither a partition nor a key is specified, Kafka selects the partition in a round-robin fashion. "partition": 1

fixedKey attribute

(Optional) Sets a fixed key to be used for all events.

"fixedKey": "myOU"

propertyKey attribute (Optional) Sets a key to be determined by looking up a property of the event content. The value is the name of the property. If the property is a map (such as the event detailMap or attributeMap), the value can be of format map.field. The property must be one that will be included in the formatted event output. If the value of the named property cannot be determined, the fixedKey is used for the event key.

"propertyKey": "imsi"

createTopic attribute (Optional) When enabled, RTI creates the topic prior to any events being sent to Kafka. By default, this attribute is disabled (false).

"createTopic": true

zookeeperServers attribute Sets the list of Zookeeper servers for the Kafka cluster. This attribute is required only if the createTopic attribute is set to true. Otherwise, this attribute is optional.

"zookeeperServers": ["localhost:2181"]

zookeeperSessionTimeout attribute (Optional) Sets the session timeout (in milliseconds) for communication with Zookeeper. This attribute is used only if the createTopic attribute is set to true. The default value is 15,000 ms (15 seconds).

"zookeeperSessionTimeout": 7500

zookeeperConnectionTimeout attribute (Optional) Sets the connection timeout (in milliseconds) for communication with Zookeeper. This attribute is used only if the createTopic attribute is set to true. The default value is 10,000 ms (10 seconds).

"zookeeperConnectionTimeout": 5000

topicPartitions attribute (Optional) Sets the number of partitions to create for a new topic. This attribute is used only if the createTopic attribute is set to true. The default value is 1.



"topicPartitions": 2

topicReplicationFactor attribute (Optional) Sets the topic replication factor for newly created topics. This is used only if createTopic is true. The default value is 1.

"topicReplicationFactor": 2

You can save processed event data to a target file periodically. Files are generated in the specified location using the stream name as the file name.


The example receives a set of attributes, specified by the selector, formats to a CSV file, and saves and compresses a file named by its stream name.


fileBasePath attribute (Required) Sets the location of the resulting file.

"fileBasePath" : "/home/rti/data"

batchSize attribute (Optional) Sets the number of records to batch save to file. The minimum batch size is set to 8192 events. The default value is 8192 batched events.

"batchSize" : 10000

batchTimeout attribute (Optional) Sets the point to save the currently batch events to file. If the batch size is breached, the batched events are flushed to file. Otherwise, the time out setting causes the event flush. The minimum batch delay is set to 15000ms (15 seconds), which is the default value.

"batchTimeout" : 30000

compress attribute (Optional) Enables or disable file compression. By default, this attribute is disabled (false).

"file": { "formatter": { "type": "CSV" }, "fileBasePath" : "/home/rti/data", "batchSize" : 1024, "batchTimeout" : 5000, "compress" : true, "compressionDelay" : 15000 }

File



"compress" : true

compressionDelay attribute (Optional) Compresses a file after a set delay. Increase this setting to reduce the number of files written, which optimizes compression. The minimum compression delay is set to 15000ms (15 seconds). The default value is15 seconds, set in milliseconds.

"compressionDelay" : 2500

This type of output enables you to define custom endpoints by a Spring Integration XML context file.



file attribute (Required) Sets the location of the Spring context file. The following example shows that the context file must define an input channel so that events from the selector or formatter can be inserted to it after processing.

"file" : "./my_spring_integration_endpoint.xml"

Note: Either a ProtocolEvent or EventViewObject can be received on the input channel depending on the selector expression if the formatter attribute is not defined.

formatter attribute (Optional) Sets which formatter to use. For example:

"formatter" : { "type" : "CSV" }

If this attribute is not provided, the input channel receives either a ProtocolEvent, a selector set to all attributes, or an EventViewObject when a set of attributes have been selected.

The following is an example of a Spring Integration file:

"spring": { "formatter" : { "type" : "CSV" }, "file" : "./my_spring_integration_endpoint.xml" }

Spring Integration



Formatter DSL The formatter section of the DSL enables you to define the type of output format. Currently OOTB supported formats include CSV and JSON. A formatter is bound to an output definition. The formatter receives either a set of attributes to format or a ProtocolEvent object. For more information, refer to Selector section.

type attribute

(Required) Sets the formatting to be used. Supported values include CSV, JSON, and CUSTOM

Custom formatting The CUSTOM formatter enables you to plug in custom developed formatting code. Formats such as parquet, Avro, BSON, and like are examples of custom formatters.

<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:int="http://www.springframework.org/schema/integration" xmlns:int-file="http://www.springframework.org/schema/integration/file" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-4.0.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd http://www.springframework.org/schema/integration/file http://www.springframework.org/schema/integration/file/spring-integration-file.xsd"> <int:channel id="input"/> <int:channel id="convertedChannel"/> <int:object-to-json-transformer id="convertProtocolEventToJsonBean" input-channel="input" output-channel="convertedChannel" auto-startup="true"/> <int-file:outbound-channel-adapter id="jsonFileOutbound" channel="convertedChannel" directory="/home/rti/data/json-example" delete-source-files="true" /> </beans>

"formatter": { "type": "CSV" }



plugin attribute (Optional) Defines custom formatting implementation. The Java class, inclusive of package and properties define the custom formatting implementation. This attribute is required only when the type is set to CUSTOM.

Java formatter plug-ins are developed by using the Component API and are deployed in the platform using RTISH. For more information, refer to the AbstractFormatter class in the Component API Java documentation and RTISH deployments in the EMC Real-Time Intelligence Administrators Guide.

Component SDK RTI Streaming provides you with a collection of APIs to build reusable custom components to use in the Streaming DSL definitions.

EMC recommends that you use standard components to implement your streams, if possible. Refer to Component libraries for more information. Use the Streams SDK to implement custom components only when the existing standard components do not provide the required functionality.

Custom components can be one of the following types:

• Filter—Accepts or rejects an event according to some criteria

• Enricher—Adds attributes to an event, often derived from reference data or by simple computation or combination of other event attributes

• Transformer—Modifies an existing attribute of an event

• Formatter—Renders the internal structure of an event or subscriber in another format, such as JSON or CSV

To implement a custom component, define three Java classes:

• Component—The component itself, which implements the logic of the enricher, transformer, filter, or formatter

• Component Specification—A Java representation of the component configuration

• Component Specification Builder—A builder for the component specification

"formatter": { "type": "CUSTOM", "plugin" : { "class": "com.mycompany.plugin.formatter.MyCustomFormatter", "properties" : { "prop1" : "value1", "prop2" : 2 } } } }


http://dcade.lss.emc.com/confluence/display/CTORTI/Filter

http://dcade.lss.emc.com/confluence/display/CTORTI/Enricher

http://dcade.lss.emc.com/confluence/display/CTORTI/Transformer

http://dcade.lss.emc.com/confluence/display/CTORTI/Formatter

http://dcade.lss.emc.com/confluence/display/CTORTI/Building+a+component

http://dcade.lss.emc.com/confluence/display/CTORTI/Defining+Component+DSL

http://dcade.lss.emc.com/confluence/display/CTORTI/Deploying+a+component


Each custom RTI Stream component must have an associated specification class that defines its configuration. See Defining a component DSL for a description of this class.

The component itself must extend the appropriate abstract component class for the component type:

• Filter—Extends io.pivotal.rti.component.AbstractFilter

• Enricher—Extends io.pivotal.rti.component.AbstractEnricher

• Transformer—Extends io.pivotal.rti.component.AbstractTransformer

• Formatter—Extends io.pivotal.rti.component.AbstractFormatter

Perform the following steps to implement the custom component:

1. Implement a default constructor and a constructor that takes the associated specification class. Ensure that you call the superclass constructors in both cases.

2. Implement an apply method. This method implements the logic of the component. The requirements for the apply method differ between the various component types.

3. Override the initialize method if custom component initialization is required. The superclass ensures that a component specification is present and initializes the reference data store, if required. Ensure that the call to super.initialize() is present in the custom component.

Accessing reference data A custom component often must access reference data to perform enrichment, filtering, or transformation. On component initialization, using the super.initialize() call, the colocated in-memory data store is looked up and the inherited refDataStore variable is set. The store name is normally obtained from the component's configuration specification.

For any custom stores for which you want to configure more than one store reference, override the super.initialize() method with the necessary code.

Library dependencies To build custom components, include the streams-api jar dependency in your project environment.

Using Gradle Include the following Gradle dependencies:

compile "io.pivotal.rti:streams-api:3.0.0-vXX"

vXX is the final release build number of the RTI package.

Building a custom RTI component


http://dcade.lss.emc.com/confluence/display/CTORTI/Filter

http://dcade.lss.emc.com/confluence/display/CTORTI/Enricher

http://dcade.lss.emc.com/confluence/display/CTORTI/Transformer

http://dcade.lss.emc.com/confluence/display/CTORTI/Formatter


Using Maven Include the following Maven dependency:

vXX is the final release build number of the RTI package.

A filter component filters events from the stream that is based on criteria that you set.

As with all custom RTI Stream components, you must define an associated configuration specification class for the component. For filter components, the specification class must extend io.pivotal.rti.component.dsl.specification.AbstractFilterSpecification.

To implement the custom filter component, extend the io.pivotal.rti.component.AbstractFilter class, and perform the following steps:

1. Implement a default constructor. The default constructor must call the superclass default constructor.

2. Implement a constructor that takes the associated specification class. This constructor must call the superclass constructor that takes an io.pivotal.rti.component.dsl.specification.ComponentSpecification.

3. Override the initialize method to configure the class that is based on the specification, if required. If an error occurs during initialization, this method throws an io.pivotal.rti.component.exception.FailedComponentInitializationException.

4. Implement is the apply method, which performs the filtering. If the event is accepted, the apply method returns true. If the event is rejected, the apply method returns false.

The following example demonstrates filtering that is based on the call type of an event. There is a dependency on class CallTypeFilterSpecification, which must implement the io.pivotal.rti.component.dsl.specification.AbstractFilterSpecification and expose a getCallType method.

<dependency> <groupId>io.pivotal.rti</groupId> <artifactId>streams-api</artifactId> <version>3.0.0-vXX</version> </dependency>

Filter component



An enricher component enriches an event with additional data before passing it on to the next stage for processing. Enrichers are typically used to add reference data to an event or to add a new attribute to the event derived from other attributes.

As with all custom RTI Stream components, you must define an associated configuration specification class for the component. For enricher components, the specification class must extend io.pivotal.rti.component.dsl.specification.AbstractEnrichmentSpecification.

To implement the custom Enricher component itself, extend the io.pivotal.rti.component.AbstractEnricher class and perform the following steps:


2. Implement a constructor that takes the associated specification class. This constructor must call the superclass constructor that takes an

package io.pivotal.rti.streaming.examples; import io.pivotal.rti.component.AbstractFilter; import io.pivotal.rti.component.exception.FailedComponentInitializationException; import io.pivotal.rti.protocols.ProtocolEvent; public class CallTypeFilter extends AbstractFilter { private long acceptedCallType; public CallTypeFilter() { super(); } public CallTypeFilter(CallTypeFilterSpecification specification) { super(specification); } @Override public void initialize() throws FailedComponentInitializationException { if (!hasInitialized) { super.initialize(); CallTypeFilterSpecification spec = (CallTypeFilterSpecification) super.specification; acceptedCallType = spec.getCallType(); } } /** * Key method to implement for custom logic. **/ public Object apply(Object obj) { if (!(obj instanceof ProtocolEvent)) { return false; } ProtocolEvent event = (ProtocolEvent) obj; Long callType = event.getProtocolDetails().get("callType", Long.class); if ((callType == null) || (callType.longValue() != acceptedCallType)) { return false; } return true; } }

Enricher component



io.pivotal.rti.component.dsl.specification.ComponentSpecification.

3. Override the initialize method to configure the class based on the specification, if required. If an error occurs during initialization, this method throws an io.pivotal.rti.component.exception.FailedComponentInitializationException.

4. Implement the apply method, which performs the enrichment. This method returns the enriched event.

The following example demonstrates event enrichment that adds a +-separated list of group names to an event that is based on a search of the event's International Mobile Subscriber Identity (IMSI) in reference data. This enrichment might be used to flag the event as belonging to a customer that is entitled to a certain class of service. There is a dependency on class GroupMembershipEnricherSpecification, which extends io.pivotal.rti.component.dsl.specification.AbstractEnrichmentSpecification and exposes a getStoreName method.



package io.pivotal.rti.streaming.examples; import java.util.Set; import io.pivotal.rti.component.AbstractEnricher; import io.pivotal.rti.component.exception.FailedComponentInitializationException; import io.pivotal.rti.gemfire.util.PdxHelper; import io.pivotal.rti.protocols.ProtocolEvent; import io.pivotal.rti.t.refdata.SubscriberProfile; public class GroupMembershipEnricher extends AbstractEnricher { public GroupMembershipEnricher() { super(); } public GroupMembershipEnricher(GroupMembershipEnricherSpecification specification) { super(specification); } @Override public void initialize() throws FailedComponentInitializationException { if (!hasInitialized) { super.initialize(); } } /** * Key method to implement for custom logic. **/ public Object apply(Object obj) { if (!(obj instanceof ProtocolEvent)) { return obj; } ProtocolEvent event = (ProtocolEvent) obj; String key = event.getPrimaryId(); // // Do a store lookup using the primaryId of the ProtocolEvent // Object value = refDataStore.get(key); if (value == null) { return obj; } // // Convert GemFire PDX type to a Pojo SubscriberProfile profile = PdxHelper.getPojoFromObject(value); StringBuilder result = new StringBuilder(); Set<String> subscriberGroups = profile.getSubscriberGroups(); boolean first = true; for (String subscriberGroup: subscriberGroups) { if (!first) { result.append('+'); } result.append(subscriberGroup); first = false; } event.getProtocolAttributeMap().put("profile.groups", result.toString()); return event; } }



This component transforms an event by changing its attributes before passing it on to the next stage for processing. Typical transformer use is to anonymize or remove personally identifying information in an event or to normalize the format or scale of event attributes.

As with all custom RTI Stream components, you must define an associated configuration specification class for the component. For transformer components, the specification class must extend io.pivotal.rti.component.dsl.specification.AbstractTransformerSpecification.

To implement the custom Transformer component itself, the developer must extend the io.pivotal.rti.component.AbstractTransformer class, and perform the following steps:


2. Implement a constructor taking the associated specification class. This constructor must call the superclass constructor that takes an io.pivotal.rti.component.dsl.specification.ComponentSpecification.

3. Override the initialize method to configure the class based on the specification, if required. If an error occurs during initialization, this method throws an io.pivotal.rti.component.exception.FailedComponentInitializationException.

4. Implement the apply method, which performs the transformation. This method returns the transformed event.

The following example demonstrates event transformation that obfuscates the destination IP address of an event, replacing routable IPs with one value defined in the specification, and non-routable IPs with another. There is a dependency on class obDestinationIPObfuscatorSpecification, which extends io.pivotal.rti.component.dsl.specification.AbstractTransformerSpecification and exposes getRoutableIP and getNonRoutableIP methods.

Transformer component



package io.pivotal.rti.streaming.examples; import java.net.InetAddress; import java.net.UnknownHostException; import com.google.common.net.InetAddresses; import io.pivotal.rti.component.AbstractTransformer; import io.pivotal.rti.component.exception.FailedComponentInitializationException; import io.pivotal.rti.protocols.ProtocolDetailMap; import io.pivotal.rti.protocols.ProtocolEvent; public class DestinationIPObfuscator extends AbstractTransformer { private String routableIP; private String nonRoutableIP; public DestinationIPObfuscator() { } public DestinationIPObfuscator(DestinationIPObfuscatorSpecification specification) { super(specification); } @Override public void initialize() throws FailedComponentInitializationException { if (!hasInitialized) { super.initialize(); DestinationIPObfuscatorSpecification spec = (DestinationIPObfuscatorSpecification) specification; routableIP = spec.getRoutableIP(); nonRoutableIP = spec.getNonRoutableIP(); } } /** * Key method to implement for custom logic. **/ public Object apply(Object obj) { if (!(obj instanceof ProtocolEvent)) { return obj; } ProtocolEvent event = (ProtocolEvent) obj; ProtocolDetailMap protocolDetails = event.getProtocolDetails(); if (protocolDetails.containsKey("destinationIp")) { String destinationIP = protocolDetails.get("destinationIp", String.class); if (InetAddresses.isInetAddress(destinationIP)) { try { if (InetAddress.getByName(destinationIP).isSiteLocalAddress()) { protocolDetails.put("destinationIp", nonRoutableIP); } else { protocolDetails.put("destinationIp", routableIP); } } catch (UnknownHostException exn) { // Do nothing } } } return obj; } }



This component formats events to a target output format such as AVRO, Parquet, JSON, CSV, and the like. The RTI platform provides OOTB formatters for JSON and CSV.

Unlike other custom RTI Stream components, custom formatters do not require that you define an associated configuration specification class.

To implement the custom formatter, extend the io.pivotal.rti.component.AbstractFormatter class, and perform the following steps:


2. Implement a constructor that takes the associated specification class. This constructor must call the superclass constructor that takes an io.pivotal.rti.component.dsl.specification.FormatterSpecification.

3. Override the initialize method to configure the class based on the specification, if required. If an error occurs during initialization, this method throws an io.pivotal.rti.component.exception.FailedComponentInitializationException..

4. Override the getType() method by returning the formatting type as a string. For example, if implementing AVRO formatting, return the string AVRO.

5. Implement the apply method, which performs the formatting. This method returns the formatted event ready to be dispatched on to target endpoints.

The apply method receives an event from the selector process which can be of type:

• ProtocolEvent—If the all wildcard (*) is used in the selector expression.

• EventViewObject—If a selection of attributes are defined as the selector expression.

The following example shows how to format events to a target output.

Formatter component



To use an implemented custom component in the Streaming DSL, a component specification class is required. This class defines a set of attributes that configure the component for the wanted behavior, which results in a JSON object that is incorporated

@ManagedResource public class CustomDomainFormatter extends AbstractFormatter { private static final Logger LOG = LoggerFactory.getLogger(CustomDomainFormatter.class); @Override public void initialize() throws FailedComponentInitializationException { try { // Perform initialization logic if required } catch (Exception e) { LOG.error("Initialisation failed.", e); throw new FailedComponentInitializationException(e); } } /** * Apply a formatting logic to the event and return the result. * * @param event the event to be process by the component either a ProtocolEvent all attributes or EventViewObject for a subset of attributes. * @return result of the process. */ @Override public Object apply(Object event) { if( event instanceof ProtocolEvent){ return format( (ProtocolEvent) event); } return format( (EventViewObject) event ); } private Object format(ProtocolEvent event){ Object formattedEvent = null; /** * Formatting code */ return formattedEvent; } private Object format(EventViewObject event){ Object formattedEvent = null; String[] fields = eventViewObject.getAttributes(); Object[] values = eventViewObject.getValues(); /** * Formatting code */ return formattedEvent; } /** * Type of formatter. * @return the supported formatting type as a string. */ @Override public String getType() { return "customFormatterType"; } }

Defining a component DSL



into a stream definition. Enricher, filter, and transformer custom components must define a specification Java class to represent the component DSL.

The specification class must extend the appropriate abstract specification class for the component type:

• Enricher—Extends io.pivotal.rti.component.dsl.specification.AbstractEnrichmentSpecification

• Filter —Extends io.pivotal.rti.component.dsl.specification.AbstractFilterSpecification

• Transformer—Extends io.pivotal.rti.component.dsl.specification.AbstractTransformerSpecification

To implement the specification class, extend the appropriate abstract specification class and perform the following actions:

• Component naming

Add a @com.fasterxml.jackson.annotation.JsonRootName annotation to the class, specifying the desired component name.

• Attribute definition

Define fields for configurable properties of the component.

Implement getters and setters for the configuration fields.

Add @com.fasterxml.jackson.annotation.JsonProperty annotations to the setter methods, indicating for each whether the field is required or not, and the name of the corresponding JSON field.

• Validation logic

Override the validate method of io.pivotal.rti.component.dsl.specification.AbstractComponentSpecification, to verify that all required fields are specified and valid. If a required field is missing, or the field contains invalid data, this method throws an io.pivotal.rti.component.exception.InvalidSpecificationException.

• Serialization

Override the fromData method of io.pivotal.rti.component.dsl.specification.AbstractComponentSpecification, to deserialize the specification from its GemFire-compatible representation.

Override the toData method of io.pivotal.rti.component.dsl.specification.AbstractComponentSpecification, to serialize the specification to its GemFire-compatible representation.



• Configuration description

Override the toString method of java.lang.Object, to provide a textual representation of the configuration that is displayed when the stream is listed in RTISH.

• Set component implementation

Override the setRtiComponentClass method, setting the rtiComponentClass field to the class of the associated custom component.

When you deploy the stream, RTI automatically instantiates the appropriate specification class by using annotations to map between the JSON elements and Java fields. The following example shows the specification class for the CallTypeFilter custom filter described in Filtering.



package io.pivotal.rti.streaming.examples; import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; import com.fasterxml.jackson.annotation.JsonIgnoreProperties; import com.fasterxml.jackson.annotation.JsonProperty; import com.fasterxml.jackson.annotation.JsonRootName; import com.gemstone.gemfire.DataSerializer; import io.pivotal.rti.component.dsl.specification.AbstractFilterSpecification; import io.pivotal.rti.component.exception.InvalidSpecificationException; // Component name @JsonRootName(value = "callTypeFilter") public class CallTypeFilterSpecification extends AbstractFilterSpecification { private long callType; public long getCallType() { return callType; } @JsonProperty(required = false, value = "callType") public void setCallType(long callType) { this.callType = callType; } /** * Required to set the associated component implementation. */ @Override public void setRtiComponentClass() { rtiComponentClass = CallTypeFilter.class; } /** * Ensures the attributes set meet the required validation criteria. */ @Override public void validate() throws InvalidSpecificationException { super.validate(); if (callType < 0) { throw new InvalidSpecificationException("The callType field must be non-negative."); } } /** * Required for the configuration persistence. */ @Override public void fromData(DataInput dataInput) throws IOException, ClassNotFoundException { super.fromData(dataInput); callType = DataSerializer.readPrimitiveLong(dataInput); } @Override public void toData(DataOutput dataOutput) throws IOException { super.toData(dataOutput); DataSerializer.writePrimitiveLong(callType, dataOutput); } /** * Used to describe a component configuration for a given stream definition. */ @Override public String toString() { StringBuilder sb = new StringBuilder(); sb.append(String.format("callTypeFilter { %n")); sb.append(String.format(SPEC_OUTPUT_FORMAT, "callType", getCallType())); sb.append(String.format("} %n")); return sb.toString(); } }



This class produces the following JSON object used as the component DSL.

Component DSL Builder Class

To support programmatic definition of a component specification, define a corresponding builder class for each specification. The builder constructs a validated specification instance, according to the builder design pattern. For more information about the builder design pattern, refer to Builder Pattern.

To implement the builder class for a specification, the developer must implement the io.pivotal.rti.component.dsl.ComponentSpecificationBuilder interface, and perform the following tasks:

1. Define fields for configurable properties of the component.

2. Implement a setter method for each field. The setter should returning the builder object, so that setter calls can be chained.

3. Implement the build method, which should construct and validate an instance of the associated specification class.

The following example shows the builder class for the CallTypeFilterSpecification defined in the above section.

"callTypeFilter" : { "callType" : 3 }

package io.pivotal.rti.streaming.examples; import io.pivotal.rti.component.dsl.ComponentSpecificationBuilder; import io.pivotal.rti.component.dsl.specification.ComponentSpecification; import io.pivotal.rti.component.exception.InvalidSpecificationException; public class CallTypeFilterSpecificationBuilder implements ComponentSpecificationBuilder { private long callType; public CallTypeFilterSpecificationBuilder setCallType(long callType) { this.callType = callType; return this; } @Override public ComponentSpecification build() throws InvalidSpecificationException { CallTypeFilterSpecification specification = new CallTypeFilterSpecification(); specification.setCallType(callType); specification.validate(); return specification; } }


https://en.wikipedia.org/wiki/Builder_pattern


Packaging Custom components are packaged in standard Java JAR files that are contained in .tar.gz archives.

The JAR file must include a text resource that lists the custom component specifications defined in the JAR. This file should be in the META-INF/services directory in the custom component project, and must be named io.pivotal.rti.component.dsl.specification.ComponentSpecification. The file must contain one line for each custom component, consisting of the fully qualified component specification class name. For the filter example that is described in Filtering, the file must contain the following line:

io.pivotal.rti.streaming.examples.CallTypeFilterSpecification

For the examples in this section, the io.pivotal.rti.component.dsl.specification.ComponentSpecification resource is a text file with the following content.

Deployment To deploy the custom component package, use the RTISH rti ext install command. For example, to deploy the package rti-streaming-examples.tar.gz, type the following command:

rtish> rti ext install --file rti-streaming-examples.tar.gz

Component libraries The component library provides the core feature of code reuse. This section details the OOTB core components provided by RTI.

The OOTB core components provided by RTI include:

• Filters

Event type

Event time

Expression based

− Spring Expression Language

− Groovy

Predicate filter

• Enrichers

Reference data enricher

Boolean attribute enricher

Number attribute enricher

io.pivotal.rti.streaming.examples.CallTypeFilterSpecification io.pivotal.rti.streaming.examples.GroupMembershipEnricherSpecification io.pivotal.rti.streaming.examples.DestinationIPObfuscatorSpecification

Deploying a component



• Transformation

Anonymization

− OOTB

− Custom

Groovy

− Dynamic code deployment

• Formatters

CSV

JSON

Custom Use the filtering components for even filtering. Filtering components include

• Event type

• Event time

• Expression-based:

Spring Expression Language

Groovy

Event type filtering and attributes Event type filtering, which is an OOTB component that is shipped with the core RTI platform, filters events that are based on the type set by the ingest grid process. For example:

The combination of parameters determines if an event is to be filtered or operated on with the stream.

type (Required) Selects the protocol type on which to filter. A lookup is performed against the protocolName attribute of the ProtocolEvent.

"type" : "adr"

subType (Optional) Narrows the filtering criteria by using the protocolType attribute in ProtocolEvent. The default value is -1, which ignores the subtype field and applies the filter to all subtypes.

"subType" : 4

"protocolFilter": { "type" : "adr", "subType": 4, "exclude": false }

Filtering



exclude (Optional) Includes or excludes the event for further processing. Set to false to include if the events meet the required criteria, otherwise set to true to exclude. The default is set to include (false).

"exclude" : true

Example: Filtering by type

The following DSL example filters all adr typed events into a reusable stream that is ready for further processing. This approach is ideal if there is a multiple set of independent streams that perform various computations. These streams subscribe to the adrRawStream event source.

Example: Exclude event filtering The following example shows filtering of a protocol with multiple subtypes. Only events of one of the subtypes is required.

Time-based filtering and attributes The eventTimeFilter component enables you to filter events that are in a date range.

from (Required) Sets the valid from date.

{ "name" : "adrRawStream", "using" : ["defaultFirehose"], "flow" : [ { "protocolFilter" : { "type" : "adr" } } ] }

{ "name" : "protocolFilterSubType", "using" : ["defaultFirehose"], "flow" : [ { "protocolFilter": { "type" : "adr", "subType": 4, "exclude": true } } ] }

{ "eventTimeFilter" : { "from": "09/15/2016", "to": "09/17/2016", "dateTimeFormat" : "MM/dd/yyyy" } }



"from" : "09/15/2016",

to (Required) Sets the valid to date.

"to" : "09/17/2016"

dateTimeFormat (Optional) Sets the date format used to set the to and from dates. This field follows Java's SimpleDateFormat rules. The default date format is MM/dd/yyyy.

"dateTimeFormat" : "MM/dd/yyyy"

The following DSL example filters all adr typed events that are in the specified date range. This approach is ideal if there is a multiple set of independent streams that perform various computations such as timed marketing campaigns. These streams subscribe to the adrSept16Stream event source.

Expression filtering and attributes This component provides the ability to define an expression using a supported language to filter events.

expression (Required) Sets the expression to be evaluated against the passed event.

"expression": "event.getProtocolDetails().get('latitude') > 0",

lang (Optional) Sets the executing language for the expression evaluation. Supported languages include Groovy, Javascript, and SpEL, the Spring Expression Language. The default language is SpEL.

"lang" : "GROOVY"

{ "name" : "adrSept16Stream", "using" : ["defaultFirehose"], "flow" : [ { "protocolFilter" : { "type" : "adr" } }, { "eventTimeFilter" : { "from": "09/15/2016", "to": "09/17/2016" } } ] }

"expressionFilter": { "expression": "event.getProtocolDetails().get('latitude') > 0", "lang" : "GROOVY" }



Spring Expression Language

By default RTI uses SpEL for dynamic runtime expression evaluation. The following example evaluates the ProtocolEvent protocolDetails attribute for a latitude that is greater than zero. You do not need to dereference the event object for the public attributes.

For more information about SpEL, refer to the documentation on the Spring website.

Groovy RTI supports the Groovy programming language for dynamical runtime expression evaluation. For complex expressions, Groovy is the preferred language because the expression is compiled to byte code for optimal evaluation performance. The event payload is provided to the Groovy context as the event variable.

For more information about Groovy, refer to the documentation on the Groovy website.

{ "name" : "spelfilteringExample", "using" : ["defaultFirehose"], "flow" : [ { "expressionFilter": { "expression": "protocolDetails['latitude'] > 0" } } ] }

{ "name" : "groovyFilterExample", "using" : ["defaultFirehose"], "flow" : [ { "expressionFilter": { "expression": "event.getProtocolDetails().get('latitude') > 0", "lang" : "GROOVY" } } ] }



Javascript RTI supports Javascript, which is similar to Groovy. The event payload is provided to the Javascript context as the event variable.

{ "name" : "javascriptFilterExample", "using" : ["defaultFirehose"], "flow" : [ { "expressionFilter": { "expression": "event.getPrimaryId() == 'success'", "lang" : "JAVASCRIPT" } } ] } }

Predicate filtering This filter enables you to use a set of filters together using AND/OR logic. Use this approach to group a set of filters in a single stream.

type (Required) Sets the logical operator to be used to evaluate the filter set. Both AND and OR are supported.

"type": "OR"

"predicate": { "type": "OR", "filters": [ { "filter": { "expressionFilter": "protocolDetails['age'] >= 18" } }, { "filter": { "expressionFilter": "protocolDetails['age'] <= 21" } } ] }



filters (Required) Sets the list of filters to evaluate. Two or more filters are required for a valid configuration.

The enricher components enable you to enrich events with colocated in-memory reference data or based on conditional logic. Enrichment components include:

• Reference data enrichment

• Value attribute enrichment

• Conditional attribute enrichment

Reference data enrichment The enricher component enables you to enrich events with colocated in-memory reference data. Enrichment value lookup uses either a primary or composite key. On a reference data match, either selected attributes can enrich the event or the whole value object can be added to the event.

The following example shows reference data lookup by using a primary key:

The following example shows reference data lookup by using a composite key:

primaryKey (Required) Takes the attribute value from the event and is used to search against the defined store for a match. This attribute is required if the compositeKey attribute is not provided

"filters": [ { "filter": { "expressionFilter": "protocolDetails['age'] >= 18" } }, { "filter": { "expressionFilter": "protocolDetails['age'] <= 21" } } ]

{ "enricher" : { "primaryKey" : "tac", "enrichAttributes" : "* 'mobileEquipment'", "storeName": "REF_MobileEquipment" } }

{ "enricher" : { "compositeKey" : { "attributes" : ["lac", "cellId"], "separator" : "_" }, "enrichAttributes" : "cellID,CellLAC,siteLatitude 'cellSiteLatitude', siteLongitude 'cellSiteLongitude'", "storeName": "REF_CellTowers" } }

Enrichment



"primaryKey" : "tac"

For example, if the event has the value 12345 for the tac attribute, the value is used to perform a look up against the defined reference data store to get the value object.

compositeKey (Required) Builds a key by concatenating attribute values with a separator and is used to perform a search against the defined store for a match. This attributed is required if the primaryKey attribute is not provided

For example, if the event has the value 12345 for the location area code (LAC) and 99999 for the cellId, the values are concatenated with the separator “_” to form this key: 12345_99999. The value is used to perform a look up against the defined reference data store to get the value object.

storeName (Required) Defines the local in-memory store on which to perform the key lookup.

"storeName": "REF_MobileEquipment"

enrichAttributes (Required) Defines a set of attributes from the returned reference data object that is used to enrich the current event. You can define attributes without event attribute dereferencing, with values being looked up in a cascading search algorithm. The search algorithm first searches for an attribute match in the details map and then in the attributes map before searching the base ProtocolEvent map attributes.

"enrichAttributes" : "cellID,CellLAC,siteLatitude 'cellSiteLatitude',siteLongitude 'cellSiteLongitude'",

This expression results in multiple inserts in the ProtocolEvent attribute map, as shown in the following example.

Use the standard select all wildcard (*) to place the looked-up value directly in the event attribute map. The following example defines a select all and sets the name of the attribute to be used in the event attribute map.

"enrichAttributes" : "* 'mobileEquipment'",

"compositeKey" : { "attributes" : ["lac", "cellId"], "separator" : "_" }

protocolAttributeMap { // Key -> Value {"cellID", cellID }, {"CellLAC", CellLAC }, {"cellSiteLatitude", siteLatitude }, {"cellSiteLongitude", siteLongitude } }



The following example shows the resulting map with this entry:

If you select the whole object for event enrichment and do not provide the attribute name, the default name that is given as the key is the store name. In the example, if you do not provide mobileEquipment, the key is set to REF_MobileEquipment in the attributes map.

The following example shows the resulting map with this entry:

Example: Device enrichment using a primary key

Example: Cell tower enrichment using a composite key

Value attribute enrichment This component enables you to enrich, or add, new value-based attributes by using a specified name and type to protocolAttributeMap. This enricher type is useful when you want to remap attribute names for metrics calculation. For example, if a pre-built component operates on a specific attribute name, but the source event uses a different name, this component can perform the required mapping to ensure that the pre-built component works as intended.

This component also enables you to calculate the sum of two value attributes and enriches with a specified name and type.

Key uses for this component include:

• Mapping an attribute to a new name for metrics calculation

• Preparing correct value types for a calculation

• Calculating the sum of two value attributes

protocolAttributeMap { // Key -> Value {"mobileEquipment", deviceObject } }

protocolAttributeMap { // Key -> Value {"REF_MobileEquipment", deviceObject } }

{ "enricher" : { "primaryKey" : "tac", "enrichAttributes" : "* 'mobileEquipment'", "storeName": "REF_MobileEquipment" } }

{ "enricher" : { "compositeKey" : { "attributes" : ["lac", "cellId"], "separator" : "_" }, "enrichAttributes" : "cellID,CellLAC,siteLatitude 'cellSiteLatitude', siteLongitude 'cellSiteLongitude'", "storeName": "REF_CellTowers" } }



The following example creates a new attribute, cellIdentifier with the value of type long, in the protocolAttributeMap. This attribute’s value comes from the value of attribute cellId.

The following example creates a new attribute, download, with the value of type long, in the protocolAttributeMap. The value of the attribute is calculated by adding respClientHeaderSize and respClientBodySize, which are attributes from the protocolDetailMap.

numberAttributeEnrichment Top level attribute that defines a list of enrichment definitions and forms a logical grouping.

"numberAttributeEnrichment" : [ {}, {}, .....]

attribute (Required) The original attribute name in ProtocolEvent that maps to a new attribute.

"attribute" : "cellId"

enrichAttribute (Required) The enriched attribute name.

"enrichAttribute" : "cellIdentifier"

type (Required) Defines the target type to which the attribute value must be cast. The value is converted to the target type, if necessary. If the value is passed as string 1234 and the required type is an integer, then the value is converted to an integer. If a value cannot be converted, an error is logged. Valid types include:

• Byte

• Double

• Float

{ "numberAttributeEnricher": { "numberAttributeEnrichment" : [ { "attribute" : "cellId", "enrichAttribute" : "cellIdentifier", "type" : "long" } ] }

{ "numberAttributeEnricher": { "numberAttributeEnrichment" : [ { "attribute" : "respClientHeaderSize + respClientBodySize", "enrichAttribute" : "download", "type" : "long" } ] }



• Integer

• Long

• Short

"type" : "long"

Conditional attribute enricher This component enables you to add Boolean flags according to logical SpEL-based expressions for analytical purposes.

The following example creates two new attributes in the protocolDetailMap:

• The http.status.success attribute is set to true if responseStatus is greater than or equal to 200 and less than or equal to 299. Otherwise, it is false.

• The http.status.fail attribute is set to true if responseStatus is greater than or equal to 400 and less than or equal to 599. Otherwise, it is false.

conditionEnrichment (Required) Defines a list of conditional enrichment definitions and forms a logical grouping.

"conditionEnrichment" : [ {}, {}, .... ]

condition (Required) Defines a Boolean expression to evaluate the passed event to true or false. Use a valid SpEL expression.

"condition" : "protocolDetails['responseStatus'] >= 200 && protocolDetails['responseStatus'] <= 299",

enrichAttribute (Required) Name of the attribute created with the evaluated Boolean value.

"enrichAttribute": "http.status.success"

{ "conditionEnricher": { "conditionEnrichment" : [ { "condition" : "protocolDetails['responseStatus'] >= 200 && protocolDetails['responseStatus'] <= 299", "enrichAttribute": "http.status.success" }, { "condition" : "protocolDetails['responseStatus'] >= 400 && protocolDetails['responseStatus'] <= 599", "enrichAttribute": "http.status.fail" } ] } }



An event transformation changes event attributes. Various methods that transform events include anonymization, Groovy scripting, and Spring flow or custom component-based filtering.

Anonymization The anonymization component anonymizes fields in an event. This component provides three supported algorithms, and enables you to implement and define custom algorithms.

algorithm (Required) Specifies which anonymization algorithm to apply to the set of defined fields.

"algorithm" : "md5"

Supported algorithms include:

• MD5

• SHA256

• SHA512

• Custom

In the first step of the anonymization process, OOTB algorithms require a seed value that is mixed into the selected attribute value. This seed value is referred to as the salt. Use the global system value, rti.salt, to set the salt. For more information about common properties, refer to the EMC Real-Time Intelligence (RTI) Administrators Guide.

You can provide a custom implementation of an anonymization algorithm by implementing the following Hasher interface.

The following example show how to use your custom anonymization to build streams after you write, test, package, and then deploy a CustomDomainAnonymizer algorithm to the RTI platform.

"anonymization": { "algorithm" : "md5", "fields": ["imsi","msisdn"] }

package io.pivotal.rti.anonymization; public interface Hasher extends IHasher { /** * A Unique string representing the algorithm this anonymizer implements * @return */ String getAlgorithm(); /** * Performs the anonymization of the data and returns the anonymized data. * @param data * @return */ byte [] hash(byte[] data); }

Transformation



For information about deploying code to the RTI platform with the rti ext install command, refer to the EMC Real-Time Intelligence (RTI) Administrators Guide.

fields (Required) List of fields to anonymize.

"fields": ["imsi","msisdn"]

Aggregation combines multiple events into one event. It is useful for correlating data from multiple feeds or joining the outputs of two different streams.

Script aggregator The script aggregator enables you to implement event aggregation logic in external script files. The script aggregator includes three methods:

• correlateEvent—Returns a key to which messages are correlated. Messages with the same correlationKey are aggregated. This method accepts the message payload and returns an object with which to correlate. The following shows an example of the method signature in Groovy:

def Object correlateEvent(Object payload){}

• mergeEvent—Merges an incoming event with the aggregation with which it is correlated. The first parameter to the method is the aggregation of previous events or null if the incoming event is the first in the aggregation. The second parameter is the incoming event payload. The method returns the updated aggregation object that is stored until the next event comes in or until the object is released downstream.

def Object mergeEvent(Object aggregateObject, Object payload){}

• releaseEvent—Determines whether to release the aggregate object downstream. The first parameter to the method is the aggregate event, the second parameter is the incoming event payload, and the third parameter is a reference to the aggregator. The aggregator interface supports a releaseMergedObject method that allows timer-based release of objects as well as event-based release of objects.

def Boolean releaseEvent(Object aggregateEvent, Object newEvent, Aggregator aggregator){}

Attributes The following example shows the script aggregator attributes:

{ "scriptAggregator": { "file": "Aggregator.groovy", "lang" : "groovy", "eventExpiration" : 1000,

"anonymization": { "algorithm" : "com.domain.anonymizer.CustomDomainAnonymizer", "fields": ["imsi","msisdn"] }

Aggregation



"releaseEventOnExpiration" : true, "correlateMethod" : "correlateEvent", "mergeMethod" : "mergeEvent", "releaseMethod" : "releaseEvent" } }

file attribute Location of the external script file that contains the aggregation implementation.

lang attribute Script language of the file. Supported languages include Groovy (default) and Javascript.

eventExperation attribute The time, in milliseconds, that an aggregation remains in storage without being released or receiving a new event. A value of 0 (default) indicates that the event never expires.

releaseEventOnExpiration attribute Determines if an expired aggregation is released or discarded. If set to true, an expired aggregation is automatically released downstream. If set to false, the event is discarded. The default value is false.

correlateMethod attribute Name of the method in the script file that performs correlations (refer to Script aggregator). The default value is correlateEvent.

mergeMethod attribute Name of the method in the script file that merges events with aggregations (refer to Script aggregator). The default value is mergeEvent.

releaseMethod attribute Name of the method in the script file that releases objects (refer to Script aggregator). The default value is releaseEvent.

Example The following examples show that an aggregator releases objects when two events come in.

The following shows the JSON file stream specification:

{ "name" : "groovyAggregator", "using" : ["defaultFirehose"], "enabled" : true, "flow" : [ { "scriptAggregator": {



"file": "groovyAggregator.groovy" } } ] }

The following shows the contents of the groovyAggregator.groovy file:

import io.pivotal.rti.component.Aggregator import io.pivotal.rti.protocols.ProtocolEvent def Object correlateEvent(ProtocolEvent event){ return event.getPrimaryId(); } def Object mergeEvent(ProtocolEvent aggregateEvent, ProtocolEvent newEvent){ if (aggregateEvent == null){ aggregateEvent = new ProtocolEvent(); aggregateEvent.setPrimaryId(newEvent.getPrimaryId()); aggregateEvent.getProtocolDetails().putAll(newEvent.getProtocolDetails()); aggregateEvent.getProtocolDetails().put("size",1); } else { String aggregateMessage = aggregateEvent.getProtocolDetails().get("message"); String newMessage = newEvent.getProtocolDetails().get("message"); aggregateEvent.getProtocolDetails().put("message",aggregateMessage + "_" + newMessage); aggregateEvent.getProtocolDetails().put("size",aggregateEvent.getProtocolDetails().get("size") + 1); } return aggregateEvent; } def Boolean releaseEvent(ProtocolEvent aggregateEvent, ProtocolEvent newEvent, Aggregator aggregator){ return (aggregateEvent != null && aggregateEvent.getProtocolDetails().get("size") >= 2); }



RTI includes additional components that do not fit the previously-described categories.

Scripting This component enables you to deploy code to the RTI cluster at runtime. Code is packaged as a script file, deployed in a stream, and compiled on stream startup on each service cluster node.

The following example shows how to deploy a Groovy script dynamically in to the system.

Basic Groovy script

Basic Javascript

file

Defines the Groovy script file to be deployed to each RTI service member process.

"file": "customGroovyCode.groovy",

The location of the file is based on the current working directory unless specified as a path to the script. For example:

"file": "/usr/rti/scripts/customGroovyCode.groovy",

method (Optional) Defines the method to be invoked in the script per event. The default script method to be called is transform.

"method" : "doSomthingInteresting"

lang (Optional) Defines the scripting language engine to be used for the passed script. Supported languages include Groovy (default) and Javascript.

"lang" : "groovy"

Note: Because you can deploy code to the system at runtime, ensure that the code is logically optimized for high frequency event data. Poorly written code adversely affects the RTI runtime system.

"scripting": { "file": "/usr/domain/usecases/customGroovyCode.groovy", "method" : "doSomthingInterestingMethod", "lang" : "groovy" }

import io.pivotal.rti.protocols.ProtocolEvent def Object doSomthingInterestingMethod(ProtocolEvent event){ event.setPrimaryId("changedByGroovy"); return event; }

function doSomthingInterestingMethod(event){ event.setPrimaryId("changedByJavascript"); return event; }

Miscellaneous components



Spring flow This component enables you to migrate existing Spring Integration flows in the distributed RTI platform or build custom complex flows. This component enables the RTI platform to use a rich Spring development ecosystem to enable you to build demanding solutions by using established powerful programming tools.

The following example shows how to deploy a Spring Integration flow to each service grid member, in the defined stream, ready for stream processing.

This following example shows how to accept a message from the input channel and pass it on to the custom transformer class with the result going to the output channel.

file (Required) Provides the name and path of the Spring context file and defines the logic of this component. The location of the file is based on the current working directory unless specified as a path to the file name.

"file": "/usr/domain/spring/usecases/domainContextFile.xml"

Spring context requirements All Spring context files require an input and an output channel so that events can be passed and received by configured components.

Note: Because you can deploy to the system at runtime, ensure that the code is logically optimized for high frequency event data. Poorly written code adversely affects the RTI runtime system. Ensure that the Spring context file has both input and output channels defined.

"springFlow" : { "file": "/usr/domain/usecases/domainContextFile.xml" }

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd"> <int:channel id="input" /> <int:channel id="output" /> <int:transformer input-channel="input" output-channel="output" ref="testTransformer" method="transform" /> <bean id="testTransformer" class="com.domain.transformer.CustomDomainTransformer" /> </beans>

<int:channel id="input" /> <int:channel id="output" />



Telecom components RTI provides telecom-based components, which include:

• Filtering

Subscriber and Location Opt In

Subscriber event attribute validity checks

Geofence filter

• Enrichment

Geofencing

− By CellID and Location Area Code (LAC)

− Entry and Exit

• Transformer

ProtocolEvent to Subscriber

• Formatters

Subscriber flat CSV

Subscriber Full CSV

Subscriber location CSV

Subscriber transaction CSV Three base filters are provided as part of the Telco library.

• Subscriber and Location Opt In

• Subscriber event attribute validity checks

• Geofence filtering

Geofence filter This component filters checks if one of the specified geofence IDs, in the list provided, is present in the event attribute map. Events that have a valid geofence ID pass the filter.

geofenceIds (Required) Defines the geofences on which to filter. Each geofence ID is specified using one of the geofencing components. Use one or more geofence IDs.

"geofenceIds" : ["geofence1", "geofence2"]

"geofenceFilter" :{ "geofenceIds" : ["geofence1", "geofence2"] }

Filtering



Opt-In filter This component performs an event optIn check, based on subscriber or location, using colocated reference data. Any event that passes the opt-In filter check is passed on to the next stage of processing.

Opt-In Types Opt-In types include:

• Data Type:

LocationOptIn—The location opt-In filtering logic is based on LAC, Cell ID, and valid event eventStartTime timestamp.

SubscriberOptIn—The subscriber opt-In filtering logic is based on IMSI, LAC, Cell ID, and valid event eventStartTime timestamp.

• List Type:

Black list—If a subscriber event matches the black list data type criteria, the event is filtered and dropped from the event stream.

White list—If a subscriber event fulfills the white list data type criteria, the event is passed on to enable further processing.

These Opt-In types can be combined to form "Location AND Subscriber" or "Location OR Subscriber" opt-Ins by using the Predicate component. For more information, refer to Location subscriber opt-in streams.

List types Opt-In filtering can be configured either as white list or black list. White list flows require events to match strict attribute criteria to enable further processing. That is, those events are opted in. Black list flows block events that match the criteria from further processing. That is, those events are not opted in.

White list A subscriber event must fulfill the white list Opt-In logic to enable further processing.

• Location

The primary key is the LAC. The event lac attribute is used to look up the relevant LocationOptIn record criteria.

On a successful LocationOptIn record retrieval, the LocationOptIn parameters, cellId, and time interval are used to determine if the event attributes cellId are included and if the eventStartTime timestamp is in the allocated time interval.

• Subscriber

The primary key is the IMSI. The event primayId attribute, set to the IMSI, is used to look up the relevant SubscriberOptIn record criteria. If this operation fails to return a record, the event is dropped from further processing.

"optInFilter": { "optInType": "LOCATION_BLACKLIST", "storeName": "OPTIN_BLACKLIST_Locations", "useStartTimeForOptin" : true }



On a successful SubscriberOptIn record retrieval, the SubscriberOptIn parameters, lac, cellId, and time interval are used to determine if the lac and cellId event attributes are included and if the eventStartTime timestamp is in the allocated time interval.

Note: If any logical operations fail, the event is dropped from further stream processing. Otherwise, the event is opted-In and can proceed to next stage of processing.

Black list A black list is the inverse of the white list logic. That is, if the event matching attributes exist in the list, then the component filters out the event from further processing. An empty black list or non-existent look up key allows events to pass through for further processing.

• Location—An event is dropped from further processing if the LocationOpt logic holds.

• Subscriber—An event is dropped from further processing if the SubscriberOpt logic holds.

Note: If any logical operation fails, the event is passed on for further processing. Otherwise, the event is dropped from the defined stream.

Filtering attributes The following lists the attributes used for the OptIn logic.

optInType (Optional) Defines the type of Opt-In logic to apply.

"optInType": "LOCATION_BLACKLIST",

Supported types include:

• LOCATION_WHITELIST (default)

• LOCATION_BLACKLIST

• SUBSCRIBER_WHITELIST

• SUBSCRIBER_BLACKLIST

storeName (Required) Defines the colocated reference data store that is used for the OptIn logic. This store must exist with the platform. The reference data loaded to the store is dependent on the required optIn requirements.

"storeName": "OPTIN_BLACKLIST_Locations"

useStartTimeForOptin (Optional) Uses the event start time as a test to determine if the event is in the time range for optin. The default value is false.

"useStartTimeForOptin" : true

Reference data Reference data is high volume and high velocity. Following an initial bulk file submission, delta files can be submitted periodically.



Subscriber optIn Subscriber opt-in data is used as a filter to determine if a subscriber event qualifies for further processing or if it is filtered out, based on IMSI values. This data holds the subscriber ID (anonymized IMSI), network locations, and time intervals.

Note: If the subscriber profile IMSI attribute is anonymized, apply the same algorithm and salt to the ?Subscriber identifier key to ensure IMSI lookups are based on the same anonymization criteria. If the anonymization salt is changed on the platform, then a complete reload of the SubscriberOptIn data is required.

Subscriber opt-In record format The following table describes the fields in each record. One record represents one subscriber opt-in.

Field number Name Description

1 Subscriber identifiers

The raw IMSI key. Defined as a numeric value.

2 LACs (Optional) The LACs where the subscriber can be tracked. A list of LACs can be provided delimited by a semi-colon (;). Defined as delimited integers or text literals. Defaults to * (all LACs).

3 Cell identifiers (Optional) The cell identifiers where the subscriber can be tracked. A list of cells can be provided delimited by a semi-colon (;). Defined as numeric or text literals. Defaults to * (all cell IDs).

4 Tags (Optional) Used for stream multiplexing. An opt-in can be forced to represent more than one stream subscription. Defined as delimited numeric integers. Defaults to *. Contact your RTI representative to use this advanced feature.

5 Time intervals (Optional) Intervals during which the subscriber can be tracked. An embedded delimited field with a semicolon (;) as the delimiter. The fields in order are: Time zone (Time Zone Database ID) Calendar flag (currently defaults to 6 and is not used) Interval tuple (start time|end time) For example:

GB;6;17:00:00.001| 18:00:00.000;18:30:00.00|19:00:00.000 Defaults to *.

The interval start and end times can also be expressed as a Java milliseconds timestamp: a decimal number, measured in milliseconds between the time and midnight, January 1, 1970 UTC.



Examples The following shows examples of valid subscriber opt-in records in CSV format that are ready for loading.

The following configuration fragment provides a Spring Data GemFire configuration for a colocated partition region. The region is linked to a subscriber parent region to ensure that incoming events that are based on IMSI as their key are directed to the primary cluster member. Partitioning provides event data-affinity, reference data colocation, and ability to scale the platform as the subscriber set increases. A key feature is that an event is routed to the primary cluster member that holds the relevant reference data, which removes any remote data reads and optimizes processing time.

For more information about configuring partition data, contact RTI support.

Supporting classes RTI provides a CSV parser class in the component library to help load this data structure.

• Class—SubscriberOptIn

• Parser—SubscriberOptInCSVParser

Location opt-In Location opt-in data is used as a filter to determine if a network event qualifies for further processing or if it is filtered out, based on location information. This data holds network locations (LACs, TACs, and cell IDs) and time intervals.

b61fe989172d6bbff33446544eaf1d28,111;2222;3333;4444,* 2c77ed56f11d122bbf547d72065af807,111;2222;3333;4444,11199;33399 90cf40e0868f42d6d439e50f1016bf95,*,* d7da17d5b5dd2a55e426d7702ae1e1d2,*,* 4b41afbd326e65ac74d3f07c10801226,*,* 1462aedaf03142f56fc7e619e9bdd1dd,*,*

<gfe:partitioned-region id="Subscriber_OPTIN_BLACKLIST" copies="1" recovery-delay="${rti.subscriberdata.recovery-delay}" startup-recovery-delay="${rti.subscriberdata.startup-recovery-delay}" total-buckets="37" disk-store-ref="refdata" concurrency-checks-enabled="true" persistent="true" statistics="true" colocated-with="SUBSCRIBERS_Partitioning_Parent"> <gfe:partition-resolver ref="subscriberIMSIPartitionResolver" /> <gfe:eviction type="HEAP_PERCENTAGE" action="OVERFLOW_TO_DISK" threshold="" /> </gfe:partitioned-region>



Location opt-In record format The following table describes fields in each record. One record represents one location opt-in.

Field Number Name Description1

1 LAC The LAC where the subscriber can be tracked. Defined as a numeric value.

2 Cell ID (Optional) The cell identifiers where the subscriber can be tracked. A list of cells can be provided delimited by a semi-colon (;). Defined as numeric or text literals. Defaults to * (all cell IDs).

3 Tags (Optional) Used for stream multiplexing. An opt-in can be forced to represent more than one stream subscription. Defined as delimited numeric integers. Defaults to *. Contact your RTI representative to use this advanced feature.

4 Time intervals (Optional) Intervals during which the subscriber can be tracked. An embedded delimited field with a semicolon (;) as the delimiter. The fields in order are: Time zone (Time Zone Database ID) Calendar flag (currently defaults to 6 and is not used) Interval tuple (start time|end time) For example:

GB;6;17:00:00.001| 18:00:00.000;18:30:00.00|19:00:00.000 Defaults to *.

Examples The following shows examples of valid location opt-in records in CSV format that are ready for loading:

The following configuration fragment provides a Spring Data GemFire configuration for a replicated region. The replicated region accesses the same location data in every RTI cluster node to stop any remote data reads and reduce the processing time.

Supporting classes RTI provides a CSV parser class in the component library to help load this data structure. For more information, refer to the JavaDoc in the io.pivotal.rti.component.telco.transformer package.

• Class—LocationOptIn

• Parser—LocationOptInCSVParser

51,11199;33399 235,* 54862,* 236,* 37470,*

<gfe:replicated-region id="Location_OPTIN_BLACKLIST" scope="distributed-ack" persistent="true" disk-store-ref="refdata" statistics="true" />



Subscriber event filter This component filters subscriber events that are deemed not to be valid. The following example publishes valid subscriber events ready for further processing.

Subscriber events are filtered based on the attributes being set.

requireIMSI (Optional) Checks if there is a valid IMSI present on the event. The primaryId is set with a non-empty value. The default value is true.

"requireIMSI" : true

requireCellId (Optional) Checks if there are CellId and LAC attributes present on the event. The CellId and LAC are set with a value greater than zero. The default value is true.

"requireCellId" : true

requireEventEndTime (Optional) Checks if there is a valid value set that is greater than zero for the eventEndTime. The default value is true.

"requireEventEndTime" : true

requireEventStartTime (Optional) Checks if there is a valid value set that is avalue greater than zero for the eventStartTime. The default value is false.

"requireEventStartTime" : true

requireDevice (Optional) Checks if there is a valid device attribute set with a value that is not null. The default value is false.

"requireDevice" : true

{ "name" : "validSubscriberEventsStream", "using" : ["defaultFirehose"], "flow" : [ { "subscriberEventFilter" : { "requireIMSI" : true, "requireCellId" : true, "requireEventEndTime" : true, "requireEventStartTime" : false, "requireDevice" : false, "requireLAC" : false, "requireRealMsisdn" : false, "requireLocation" : false, "requireExtTags" : false } } ] }



requireLAC (Optional) Checks if there is a valid LAC attribute with a value greater than zero. The default value is false.

"requireLAC" : true

requireRealMsisdn (Optional) Checks if there is a realMsisdn attribute present that is non-empty on the event. The default value is false.

"requireRealMsisdn" : true

requireLocation (Optional) Checks if there are longitude and latitude attributes present with values that are not set as 90 degrees latitude and 180 degrees longitude. The default value is false.

"requireLocation" : true

requireExtTags (Optional) Checks if there is an extTags attribute present whose value is a non-empty integer array present. The default value is false.

"requireExtTags" : true

Two core geofence filters are provided as part of the Telco library.

• Using CellId and LAC

• Entry and Exit attributes

Entry exit geofence enricher This component tracks events that enter or exit a geofence. It uses the results of the geofenceByCellIdEnricher, which must be upstream for the enricher to work. For every primaryId, the enricher maintains a list of geofences in which the primaryID currently is. When an event enters a new geofence, the geofence ID is added to the entry list in the event attribute map. When an event is no longer in a geofence, the geofence ID is added to the exit list in the event attribute map.

entryEnrichKey (Optional) Defines the key in the attributeMap in which the entry geofences are placed. The default value is entryGeoFences.

"entryEnrichKey" : "entryGeofences"

exitEnrichKey (Optional) Defines the key in the attributeMap in which the exit geofences are placed. The default value is exitGeoFences.

"exitEnrichKey" : "exitGeofences"

"entryExitGeofenceEnricher" :{ "entryEnrichKey" : "entryGeofences", "exitEnrichKey" : "exitGeofences" }

Enrichment



GeofenceByCellId enricher The GeofenceByCellId component enables you to enrich events with matched geofences based on the passed CellId. Define the geofences by using WKT polygons in the GeofenceByCellId DSL. Any matched geofences are added to the event's attribute map using the geofence Id.

enrichKey (Required) Defines the attribute to add to the event detail map when an event has matched the specified geofence. The attribute is used later for further processing such as filtering.

"enrichKey": "geofences",

geofences (Required) Defines a non-empty list of geofences for event matching.

id (Required) Defines the identification for the geofence geometry.

"id" : "geofence1",

geometry (Required) Defines a WKT polygon geometry as a geofence.

"geometry" : "POLYGON ((5 5, 5 25, 25 25, 25 5, 5 5))"

A single transformer is provided as part of the Telco library.

"geofenceByCellIdEnricher" : { "enrichKey": "geofences", "geofences": [ { "id" : "geofence1", "geometry" : "POLYGON ((0 0, 0 10, 10 10, 10 0, 0 0))" }, { "id" : "geofence2", "geometry" : "POLYGON ((5 5, 5 25, 25 25, 25 5, 5 5))" } ] }

"geofences": [ { "id" : "geofence1", "geometry" : "POLYGON ((0 0, 0 10, 10 10, 10 0, 0 0))" }, { "id" : "geofence2", "geometry" : "POLYGON ((5 5, 5 25, 25 25, 25 5, 5 5))" } ]

Transformers



ProtocolEvent to Subscriber component This component converts a Datacast-based ProtocolEvent into a Subscriber object. This conversion is required if the event is to calculate metrics and for telco specific formatters.

includeDetail (Optional) Includes the computed attributes in the event detail map. The default value is true.

"includeDetail" : true,

includeAttributes (Optional) Includes the core Datacast attributes held in the ProtocolEvent. The default value is true.

"includeAttributes" : true,

includeInfoElements (Optional) Includes the Informational elements in the resulting Subscriber object. Set this attribute to false if only using the resulting object for metric calculations. The default value is true.

"includeInfoElements" : true

Note: If you use ProtocolEvent in a stream that has an endpoint component and use Subscriber in another stream that has an endpoint component, a conflict occurs. This conflict leads to some attributes being unavailable from the output. After the ProtocolEvent to Subscriber component is used, Subscriber is the event payload. Make sure that all streams with endpoint component use this ProtocolEventToSubscriber component at the end of the flow definition before the event is sent to the endpoint component.

Many of the built-in RTI Telco formatters are available to be used through the Streaming API. For more information about output format for event streams, refer to the EMC Real-Time Intelligence (RTI) Administrators Guide.

"protocolEventToSubscriber": { "includeDetail" : true, "includeAttributes" : true, "includeInfoElements" : true }

Formatters



Available formatters, which you can configure through the Streaming API, include:

• Subscriber flat CSV

• Subscriber Full CSV

• Subscriber location CSV

"formatter": { "type": "CUSTOM", "plugin": { "class": "io.pivotal.rti.component.telco.formatters.SubscriberFlatCSVFormatter" } }

"formatter": { "type": "CUSTOM", "plugin": { "class": "io.pivotal.rti.component.telco.formatters.SubscriberFullCSVFormatter", "properties": { "includeAttributes" : true, "includeDetails" : true, "includeMobileEquipment" : true, "includeSubscriberProfile" : true } } }

"formatter": { "type": "CUSTOM", "plugin": { "class": "io.pivotal.rti.component.telco.formatters.SubscriberLocationCSVFormatter", "properties": { "eventTimeAsUnixEpoch" : true, "geoCoordinatesMode" : true } } }



• Subscriber transaction CSV

The Streaming API provides the functionality required for Event Stream Subscriptions (ESS). By using the Streaming API, you do not have to run a Provisioning Manager (PM) instance or employ a RabbitMQ client to send a series of AMQP messages to the PM.

Deploying a single stream The following is an example stream that replaces an ESS. This stream uses several OOTB components described in this document:

"formatter": { "type": "CUSTOM", "plugin": { "class": "io.pivotal.rti.component.telco.formatters.SubscriberTransactionCSVFormatter", "properties": { "eventTimeAsUnixEpoch" : true, "geoCoordinatesMode" : true, "geoCoordinatesRadii90" : true, "enableStatsInfo" : true, "enableTagInfo" : true, "extendedTranFormat" : true, "extTagInfo" : true } } }

Event Stream Subscription (ESS)



Attributes The following lists the attributes used for this stream definition.

using attribute Defines the inputs for this stream. landsStream provides events that are opted in by both location and subscriber, which is the same as for ESS for events.

eventTimeFilter attribute Only allows events that are in the from and to time range, which is the same as the valid date feature in ESS.



geofenceByCellIdEnricher attribute Where all the geofences are defined. If an event falls in a certain geofence, that event is enriched with the geofence ID. This method replaces the ESS method of creating geofences with separate messages. Everything is defined in one stream specification.

geofenceFilter attribute Replaces the ESS method of linking geofences to subscriptions. If an event is not enriched with a geofenceID specified in the filter, it does not proceed down the flow.

amqp attribute Contains the information required to send the events that have reached the end of the stream to AMQP. ESS required you to define org units so that each org unit had its own routingKey. Org units are no longer required because you can specify the routingKey with this attribute.

Deploying streams remotely by using the REST application Deploy streams from a remote machine if it can access a REST application that is connected to the distribution grid. The following RTISH commands set up a REST application:

To deploy this stream, post the stream definition to this REST URL: <hostAddress>:<port>/rti-t/streams/deploy

The following example uses the curl command:

This feature of the streaming API eliminates the need for the provisioning manager and the complex handshaking required to setup an ESS subscription over AMQP.



Stream management Manage streams by using the RTISH management tool.

Deploy a stream Stream deployment requires a stream.json definition file and connection to the services cluster. The following command deploys, or adds, the stream across the cluster. At system restart, all deployed and enabled streams start up and are ready for processing.

List deployed streams To list all deployed streams in the system and their status, use the following command:

Enable a stream The following command enables a stream to start processing real-time events.

Disable a stream The following command disables a stream from processing real-time events

Note: If a stream is disabled at system restart, the stream is not started for event processing.

Describe a stream The describe command shows a stream’s composition. It lists all subscriptions, flow components and endpoints to which the stream dispatches. Specify the stream’s name parameter.

# Connect to the cluster rtish> rti system connect --host localhost --port 10335 Connected to: localhost[10335]. # Deploy a distributed stream rtish> rti stream deploy --definition customStream.json Deployed customStream to system. Starting up for processing. customStream successfully started.

# List deployed streams rtish> rti stream list Name Active -------------------------------------------------------------- customStream true anotherStream false

# Enable a distributed stream rtish> rti stream enable --name customStream customStream successfully enabled.

# Disable a distributed stream rtish> rti stream disable --name customStream customStream successfully disabled.

Managing streams with RTISH



Undeploy a stream The undeploy command removes the stream across the cluster and from the system. Undeploying a stream requires a valid stream name and connection to the services cluster. Specify a valid stream name for the name parameter.

Reload a stream The reload command undeploys and redeploys a stream to its original state. Specify a valid stream name for the name parameter. Set an enabled bit to determine if the stream is enabled when it is reloaded.

# Describe a distributed stream rtish> rti stream describe --name customStream Attribute Value -------------------------------------------------------------------------------------------- name customStream enabled true using [defaultFirehose] cloneEvent false queueSize 1000 workerThreads 1 Flow - ProtocolFilter { type domainType subType -1 exclude false } Endpoint - select * selectType CASCADE_HASH type AMQP - Formatter - type json encoding UTF-8 exchange STREAM.CUSTOM routingKey MY.DOMAIN.KEY batchSize 250 batchTimeout 200 Setting - username guest password guest hostAddresses [localhost:5672] virtualHost / connectionTimeoutMillis 0 heartbeatSeconds 10 channelCacheSize 25

# Undeploy a distributed stream rtish> rti stream undeploy --name customStream customStream successfully undeployed.

# Reload a distributed stream rtish> rti stream reload --name customStream --enabled false Reloaded customStream using default definition in disabled state.



You can manage streams remotely through the rest-api application.

Create the rest-api application Create the REST API application through RTISH. The locator to which the REST API application connects determines with which grid it will work. Currently, streams only work in the distribution grid.

Note: The default port is set to 8080. Because port 8080 is often used by web applications, EMC recommends that you change the server's port number.

Deploy a stream Stream deployment requires a valid stream.json definition file. The following command deploys, or adds, the stream across the distributed cluster. At system restart, all deployed and enabled streams start up and are ready for processing.

The /rti-t/streams/deploy REST URL expects a multipart/form-data ContentType. This content type allows multiple files to be uploaded at once, which is needed if the stream specification refers to external configuration files. The stream parameter must refer to the stream specification.

The following commands deploy a stream to a cluster.

Deploy the stream definition file in JSON format and supporting files that are used by the stream definition, such as Groovy or Spring Integration configuration files. For example, deploy the following as a single atomic unit:

• The exampleEnricherStream.json stream:

//create the rest-api application rti create application --type rest-api --name rest-app //enable the rest-api application rti enable instance --instances rest-app@* //connect to the dist-grid locator rti set property --instances rest-app@* --name rti.gemfire.locators --value 127.0.0.1[10335] //set the port for the rest-api rti set property --instances rest-app@* --name server.port --value 49492

# Rest URL <hostAddress>:<port>/rti-t/streams/deploy # CURL command curl -F [email protected] localhost:9001/rti-t/streams/deploy # Response usecaseStream successfully started.

Managing streams with the REST API



• A supporting Spring Integration file named exampleEnricher.xm:

When you use the curl command to deploy the stream, stream identifies the stream definition file and exampleEnricher.xml identifies the supporting file, which is the file named defined in the exampleEnricherStream stream.

List deployed streams To list all deployed streams in the system and their status, use the following command:

{ "name" : "exampleEnricherStream", "using" : ["lorsStream"], "flow" : [ { "springFlow" : { "file": "exampleEnricher.xml" } }, { "protocolEventToSubscriber": { "includeAttributes" : true, "includeDetail" : true } } ] }

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd"> <int:channel id="input" /> <int:channel id="output" /> <int:enricher input-channel="input" output-channel="output"> <int:property name="protocolAttributeMap['sms.mo.2g.success']" expression="1" /> </int:enricher> </beans>

# Rest URL <hostAddress>:<port>/rti-t/streams/deploy --- NEED THE FULL COMMAND HERE --- # CURL command curl -F [email protected] -F [email protected] localhost:9001/rti-t/streams/deploy # Response exampleEnricherStream successfully started.



Enable a stream The following command enables a stream to start processing real-time events. Specify a valid stream name for the name parameter.

Disable a stream The following command disables a stream from processing real-time events. Specify a valid stream name for the name parameter.

Describe a stream The describe command shows a stream’s composition. It lists all subscriptions, flow components and endpoints to which the stream dispatches. Specify a valid stream name for the name parameter.

# Rest URL <hostAddress>:<port>/rti-t/streams/list # CURL command curl -X POST 'localhost:9001/rti-t/streams/list' # Response Name Active ------------------------------------------------------------- exampleEnricherStream true

# Enable a distributed stream # Rest URL <hostAddress>:<port>/rti-t/streams/enable?name=<stream-name> # CURL command curl -X POST 'localhost:9001/rti-t/streams/enable?name=<stream-name>' # Response exampleEnricherStream successfully enabled.

# Disable a distributed stream # Rest URL <hostAddress>:<port>/rti-t/streams/disable?name=<stream-name> # CURL command curl -X POST 'localhost:9001/rti-t/streams/disable?name=<stream-name>' # Response exampleEnricherStream successfully disabled.



Undeploy a stream The undeploy command removes the stream from the cluster and from the system. Specify a valid stream name for the name parameter.

Reload a stream This reload command undeploys and redeploys a stream to its original state. Specify a valid stream name for the name parameter. Set an enabled bit to determine if the stream is enabled when it is reloaded.

# Rest URL <hostAddress>:<port>/rti-t/streams/describe?name=exampleEnricherStream # CURL command curl -X POST 'localhost:9001/rti-t/streams/describe?name=exampleEnricherStream' # Response Attribute Value -------------------------------------------------------------------------------------------- name exampleEnricherStream enabled true using [lorsStream] cloneEvent false queueSize 1000 workerThreads 1 Flow - springFlow - File exampleEnricher.xml ProtocolEventToSubscriber { includeAttributes true includeDetail true includeInfoElements true } true}

# Rest URL <hostAddress>:<port>/rti-t/streams/undeploy # CURL command curl -X POST 'localhost:9001/rti-t/streams/undeploy?name=exampleEnricherStream' # Response exampleEnricherStream successfully underplayed

# Rest URL <hostAddress>:<port>/rti-t/streams/reload?name=<stream-name> # CURL command curl \-X POST 'localhost:9001/rti-t/streams/reload?name=<stream-name>'



Monitoring with JMX Streams exposes many of their attributes with JMX, such as the number of events processed, the size of threadpool queues, and configuration values. Each stream has its own package name that starts with io.pivotal.rti.streams, as shown in Figure 7. This package contains all the JMX attributes and operations for that stream.

Figure 7. io.pivotal.rti.streams example

Monitoring streams



CustomStream MBean The CustomStream MBean exposes the input and output counts for the entire stream. In the following adrStream, the input and output counts are different because the stream filters out some events, as shown in Figure 8.

Figure 8. CustomStream MBean example



JmxThreadPoolTaskExecutor MBean The JmxThreadPoolTaskExecutor MBean exposes the queue size, capacity, and active threads for each threadpool, as shown in Figure 9. If a queue's size is equal to its capacity, then this stream might slow down the system.

Figure 9. JmxThreadPoolTaskExecutor MBean example



StreamComponent MBeans Based on the contents of the stream's flow, JMX exposes different Mbeans for each component in the stream. What JMX exposes depends on the actual component.

Figure 10 is an example of a ProtocolFilter component's MBean.

Figure 10. ProtocolFilter MBean example


Chapter 4: Examples

Chapter 4 Examples


Event anonymization ........................................................................................ 96

Dynamic code deployment ............................................................................... 98

Geofence detection ......................................................................................... 100

Location subscriber opt-in streams............................................................... 104

Spring stream flow .......................................................................................... 109

Spring Integration endpoint ........................................................................... 111

Multiple endpoints .......................................................................................... 115

Event enrichment ............................................................................................ 119

Telco event enrichment .................................................................................. 123

Number attribute enrichment ......................................................................... 134

Condition enrichment ..................................................................................... 137

Component SDK.............................................................................................. 139

Kafka output .................................................................................................... 145

Event Stream Subscription (ESS) .................................................................. 147

Non-telco streams ........................................................................................... 151

KPI stores ........................................................................................................ 152


Chapter 4: Examples

Event anonymization The anonymization component enables you to anonymize sensitive fields in the protocol event.

The md5AnonymizedStream.json script is in the $RTI_HOME/examples/streaming/anonymization directory. The following example stream anonymizes the primaryId, imsi, serviceId and msisdn fields in the protocolDetails map using the md5 algorithm. The value of imsi and msisdn is encrypted after the enrichment.

Sample data The following example shows the sample data for this stream, which is in $RTI_HOME/examples/streaming-api/sampleData/md5AnonymizedStreamSampleData.json.

{ "name": "md5AnonymizedStream", "using": ["defaultFirehose"], "flow": [{ "anonymization": { "algorithm": "md5", "fields": ["primaryId", "imsi", "serviceId", "msisdn"] } }], "endpoint": { "selector": { "expression": "*" }, "outputs": [{ "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/md5AnonymizedStream" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "EventAnonymization.STREAM", "routingKey": "EventAnonymization.STREAM", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled": true }

Example stream


Chapter 4: Examples

Output event The following shows the output event with the anonymized fields. The event is written to $RTI_HOME/var/data/distgrid-one/md5AnonymizedStream as well as to the AMQP endpoint.

Custom anonymization You can write your own anonymization algorithm. For information about custom anonymization implementation, refer to the EMC Real-Time Intelligence (TRI) Developer’s Guide.

The following example stream anonymizes the imsi and msisdn fields in the protocolDetails map with a custom algorithm that is implemented in com.domain.anonymizer.CustomDomainAnonymizer. The value of imsi and msisdn is encrypted according to the algorithm in com.domain.anonymizer.CustomDomainAnonymizer after the enrichment.

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"359254061234567","eventStartTime":1425556331,"protocolDetails":{"imsi":"234158013224667","msisdn":"447795470718","mccmnc":"23415","lac":"178","cellId":"20108"}}

9b1ab200-9b91-11e6-02ad-8660a71acdf2,31fc4f8508b7b02d5f1dca4c672d2b83,e4ddfc88104e61e9938da20621b5a1eb,,3592540612345 67,1477496035616,1425556331,0,0_0,178_20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;msisdn=e4ddfc88104e61e9938da2 0621b5a1eb;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realImsi=;mccmnc=23415;longitude=180.0;imsi=31fc 4f8508b7b02d5f1dca4c672d2b83;latitude=90.0;firstLac=0;realImei=;],[]


Chapter 4: Examples

Dynamic code deployment Use the Groovy programming language to deploy new code across the node without a restart. Specify an external file that contains the Groovy code, as well as a method to call in the file. The argument passed to the method is the event payload.

The following shows the example stream $RTI_HOME/examples/streaming-api/dynamicCodeDeployment/addCellIdAndLacExampleStream.json that refers to addCellIdAndLac.groovy, which is the dynamic code file.

{ "name" : "customerAnonymizedStream", "using" : ["defaultFirehose"], "flow" : [ { "anonymization": { "algorithm" : "com.domain.anonymizer.CustomDomainAnonymizer", "fields": ["imsi", "msisdn"] } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "amqp": { "formatter": { "type": "CSV" }, "exchange": "EventAnonymization.STREAM", "routingKey": "EventAnonymization.STREAM", "batchSize": 1000, "batchTimeout": 250 } } ] }}

Example stream


Chapter 4: Examples

{ "name" : "addCellIdAndLacExampleStream", "using" : ["defaultFirehose"], "flow" : [ { "scripting": { "method":"transform", "file": "./examples/streaming-api/dynamicCodeDeployment/addCellIdAndLac.groovy" } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/dynamicCodeDeployment" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }


Chapter 4: Examples

Groovy file example

Sample data Pass the sampleData.json event through and view the output event with the cellIdAndLac attribute as shown in the following section.

Output event Passing the sample event through the system results in the following output to a CSV file in the $RTI_HOME/var/data/distgrid-one/dynamicCodeDeployment directory, and to the AMQP endpoint.

Geofence detection RTI enables you to approximate the location of a mobile device by using the connected cell tower.

The following scripts provide a marketing campaign example that defines uses cases based on geolocation. The scripts are in the $RTI_HOME/examples/streaming-api/geofenceDetection directory.

The marketingCampaignStream.json script defines two geofences of interest and then sets a filter to ensure only those geofence events are available to subscribed streams.

import io.pivotal.rti.protocols.ProtocolEvent import io.pivotal.rti.protocols.ProtocolDetailMap def Object transform(ProtocolEvent event){ ProtocolDetailMap pMap = event.getProtocolDetails() cellId = pMap.get("cellId") lac = pMap.get("lac") pMap.put("cellIdAndLac", cellId + "_" + lac ) return event }

d507f5a0-9bea-11e6-02ad-8660a71acdf2,234158013224667,447795470718,,359254061234567,1477534358009,1425556331,0,0_0,178 _20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellIdAndLac=20108_17 8;cellId=20108;realImsi=;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[]

Example stream


Chapter 4: Examples

The eastMarketingCampaignStream.json stream definition subscribes to the marketingCampaignStream stream and restricts event processing that is based on the east_region geofence.

{ "name" : "marketingCampaignStream", "using" : ["defaultFirehose"], "flow" : [ { "geofenceByCellIdEnricher" : { "geofences": [ { "id" : "west_region", "geometry" : "POLYGON ((-1 51, -1 52, 0 52, 0 51, -1 51))" }, { "id" : "east_region", "geometry" : "POLYGON ((0 51, 0 52, 1 52, 1 51, 0 51))" } ] } }, { "geofenceFilter" :{ "geofenceIds" : ["west_region","east_region"] } } //Processing on events coming in from celltowers in the west_region and east_region ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/geoFenceDetection" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }


Chapter 4: Examples

Reference data The geofences defined in the stream specification contain cell towers. This cell tower data is loaded as part of the reference data set when RTI is installed on the image. The following shows the reference data, which is in $RTI_HOME/examples/streaming-api/referenceData/cell_tower.csv.

{ "name" : "eastMarketingCampaignStream", "using" : ["marketingCampaignStream"], "flow" : [ { "geofenceFilter" :{ "geofenceIds" : ["east_region"] } } //Processing on events only in the east_region ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/geoFenceDetection" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }

#CELL_ID,CELL_NAME,CENTROID_EASTING,CENTROID_NORTHING,SITE_POSTCODE,SITE_LATITUDE,SITE_LONGITUDE,SITE_EASTING,SITE_NORTHING,RADII_90,LAC 20108,G730713 ,381642,218082,GL15 5AF,51.8515881255,-2.24084272084,383413,217076,2500,178 200,WEST_CELL_TOWER,45639,973547,GL15 5AF,51.8515881255,-0.24084272084,25378,983765,2500,100 400,EAST_CELL_TOWER,74529,246398,GL15 5AF,51.5,0.5,67395,24358,2500,300


Chapter 4: Examples

Sample data The following example shows the sample data for this stream. It is in geofenceDetectionSampleData.json in the sampleData directory.

Output event Output events are written to the $RTI_HOME/var/data/distgrid-one/geofenceDetection directory, as well as to AMQP.

Both these streams process events and write to file and AMQP. The following example shows events that are processed by the marketingCampaignStream. Events with Lac and Cell Id values that match the cell towers in the two geofences that are defined by the stream are first enriched with the containedGeoFences tag and name of the geofence in which they are found. Then, only those events with containedGeoFences values that match the filter are passed through. In this example, two of the three sample events flow through.

The following example shows the event that passes through the eastMarketingCampaignStream, which filters out any event that is not in the east_region geofence. The event is output to a file in the $RTI_HOME/var/data/distgrid-one/geofenceDetection directory, as well as to the AMQP endpoint.

One can easily verify that all three of the above sample events pass through the allData stream unfiltered if you have that stream running as well.

{"protocolName":"adr","primaryId":"11111","serviceId":"5555","sourceId":"359254061234567","eventStartTime":1425556331,"protocolDetails":{"lac":"100","cellId":"200"}} {"protocolName":"adr","primaryId":"22222","serviceId":"6666","sourceId":"1234567890","eventStartTime":1425556331,"protocolDetails":{"lac":"300","cellId":"400"}} {"protocolName":"adr","primaryId":"33333","serviceId":"7777","sourceId":"876543210","eventStartTime":1425556331,"protocolDetails":{"lac":"178","cellId":"20108"}}

dba2c5f0-9caa-11e6-02ad-8660a71acdf2,22222,6666,,1234567890,1477616832463,1425556331,0,0_0,300_400,adr,0,0,,[extTags= [];status=0;firstCellId=0;containedGeoFences=[east_region];realTarget=;lac=300;realMsisdn=;radii90=-1.0;cellId=400;re alImsi=;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[containedGeoFences=[east_region];] dba229b0-9caa-11e6-02ad-8660a71acdf2,11111,5555,,359254061234567,1477616832458,1425556331,0,0_0,100_200,adr,0,0,,[ext Tags=[];status=0;firstCellId=0;containedGeoFences=[west_region];realTarget=;lac=100;realMsisdn=;radii90=-1.0;cellId=2 00;realImsi=;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[containedGeoFences=[west_region];]

dba2c5f0-9caa-11e6-02ad-8660a71acdf2,22222,6666,,1234567890,1477616832463,1425556331,0,0_0,300_400,adr,0,0,,[extTags= [];status=0;firstCellId=0;containedGeoFences=[east_region];realTarget=;lac=300;realMsisdn=;radii90=-1.0;cellId=400;re alImsi=;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[containedGeoFences=[east_region];]


Chapter 4: Examples

Note: Deploy the marketingCampaignStream.json file first. Then, deploy the eastMarketingCampaignStream.json file, as it uses the former stream. Similarly when you undeploy streams, undeploy the dependent stream first.

Location subscriber opt-in streams A common telco use case is to have location and subscriber opt-in filters in either an AND or OR predicate. You can whitelist or blacklist both opt-In filters.

• A whitelisted filter means that an event's primaryId must exist in the region to

pass the filter.

• A blacklisted filter means that an event's primaryId must not exist in the region to pass the filter.

The following blacklist example shows that the stream provides all events that are not blacklisted by both the location and subscriber filter. The stream file is in $RTI_HOME/examples/streaming-api/optinStreams/LANDSOptInBlacklistStream.json.

Location AND Subscriber OptIn


Chapter 4: Examples

Reference data Reference data for both location and subscriber opt-ins are loaded from the location_optin.csv and subscriber_optin.csv files as shown below (found in $RTI_HOME/examples/streaming-api/referenceData).

location_optin.csv

{ "name" : "LANDSOptInBlacklistStream", "using" : ["defaultFirehose"], "flow" : [ { "predicate": { "type": "AND", "filters": [ { "optInFilter": { "optInType": "SUBSCRIBER_BLACKLIST", "storeName": "OPTIN_BLACKLIST_Subscribers" } }, { "optInFilter": { "optInType": "LOCATION_BLACKLIST", "storeName": "OPTIN_BLACKLIST_Locations" } } ] } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/optInStreams" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }


Chapter 4: Examples

300,*,*,*,* 178,*,*,*,*

subscriber_optin.csv

Sample data The following example shows the sample data for this stream. The optInStreamsSampleData.json file, which is in the $RTI_HOME/examples/streaming-api/sampleData directory, has four events that demonstrate the capability of the opt-in filters and predicates.

Output event The output events for these streams are in the $RTI_HOME/var/data/distgrid-one/optInStreams directory, as well as the AMQP endpoint. The following example shows that only the one event which is not blacklisted and filtered out by both filters is allowed to pass.

The following blacklist example shows that the stream provides all events that are not blacklisted by the location or the subscriber filter. This stream file is LORSOptInBlacklistStream.json.

33333,*,*,*,* 44444,*,*,*,*

{"protocolName":"adr","primaryId":"11111","serviceId":"5555","sourceId":"359254061234567","eventStartTime":1425556331,"protocolDetails":{"lac":"100","cellId":"200"}} {"protocolName":"adr","primaryId":"22222","serviceId":"6666","sourceId":"1234567890","eventStartTime":1425556331,"protocolDetails":{"lac":"300","cellId":"400"}} {"protocolName":"adr","primaryId":"33333","serviceId":"7777","sourceId":"876543210","eventStartTime":1425556331,"protocolDetails":{"lac":"500","cellId":"600"}} {"protocolName":"adr","primaryId":"44444","serviceId":"8888","sourceId":"625348406","eventStartTime":1425556331,"protocolDetails":{"lac":"178","cellId":"20108"}}

0d627fb0-9d12-11e6-02ad-8660a71acdf2,11111,5555,,359254061234567,1477661154091,1425556331,0,0_0,100_200,adr,0,0,,[rad ii90=-1.0;extTags=[];cellId=200;realImsi=;status=0;firstCellId=0;realTarget=;longitude=180.0;latitude=90.0;firstLac=0 ;realImei=;lac=100;realMsisdn=;],[]

Location OR Subscriber OptIn


Chapter 4: Examples

Reference data Reference data for both location and subscriber opt-ins are loaded from the location_optin.csv and subscriber_optin.csv files as shown below (found in $RTI_HOME/examples/streaming-api/referenceData).

location_optin.csv

{ "name" : "LORSOptInBlacklistStream", "using" : ["defaultFirehose"], "flow" : [ { "predicate": { "type": "OR", "filters": [ { "optInFilter": { "optInType": "SUBSCRIBER_BLACKLIST", "storeName": "OPTIN_BLACKLIST_Subscribers" } }, { "optInFilter": { "optInType": "LOCATION_BLACKLIST", "storeName": "OPTIN_BLACKLIST_Locations" } } ] } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/optInStreams" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }


Chapter 4: Examples

300,*,*,*,* 178,*,*,*,* subscriber_optin.csv

Sample data The following example shows the sample data for this stream. The optInStreamsSampleData.json file, which is in the $RTI_HOME/examples/streaming-api/sampleData directory, has four events that demonstrate the capability of the opt-in filters and predicates.

Output event Any event that is blacklisted and filtered out by both opt-in filters is not allowed to pass. The output events for this streams is written to a file in the $RTI_HOME/var/data/distgrid-one/optInStreams directory, as well as to the AMQP endpoint.

33333,*,*,*,* 44444,*,*,*,*

{"protocolName":"adr","primaryId":"11111","serviceId":"5555","sourceId":"359254061234567","eventStartTime":1425556331,"protocolDetails":{"lac":"100","cellId":"200"}} {"protocolName":"adr","primaryId":"22222","serviceId":"6666","sourceId":"1234567890","eventStartTime":1425556331,"protocolDetails":{"lac":"300","cellId":"400"}} {"protocolName":"adr","primaryId":"33333","serviceId":"7777","sourceId":"876543210","eventStartTime":1425556331,"protocolDetails":{"lac":"500","cellId":"600"}} {"protocolName":"adr","primaryId":"44444","serviceId":"8888","sourceId":"625348406","eventStartTime":1425556331,"protocolDetails":{"lac":"178","cellId":"20108"}}

c31f7650-9d12-11e6-02ad-8660a71acdf2,11111,5555,,359254061234567,1477661458997,1425556331,0,0_0,100_200,adr,0,0,,[rad ii90=-1.0;extTags=[];cellId=200;realImsi=;status=0;firstCellId=0;realTarget=;longitude=180.0;latitude=90.0;firstLac=0 ;realImei=;lac=100;realMsisdn=;],[] c320d5e0-9d12-11e6-02ad-8660a71acdf2,22222,6666,,1234567890,1477661459006,1425556331,0,0_0,300_400,adr,0,0,,[radii90= -1.0;extTags=[];cellId=400;realImsi=;status=0;firstCellId=0;realTarget=;longitude=180.0;latitude=90.0;firstLac=0;real Imei=;lac=300;realMsisdn=;],[] c321e750-9d12-11e6-02ad-8660a71acdf2,33333,7777,,876543210,1477661459013,1425556331,0,0_0,500_600,adr,0,0,,[radii90=- 1.0;extTags=[];cellId=600;realImsi=;status=0;firstCellId=0;realTarget=;longitude=180.0;latitude=90.0;firstLac=0;realI mei=;lac=500;realMsisdn=;],[]


Chapter 4: Examples

Spring stream flow The Spring flow allows injection of Spring Integration into the stream.

The following shows the example stream $RTI_HOME/examples/streaming-api/springStreamFlow/springIntegrationStream.json, which refers to the customStream.xml file.

The springFlow file attribute refers to a separate Spring Integration configuration file, which is loaded when the stream is deployed. The spring configuration file must include a channel named input and a channel named output. When you deployed with RTISH, it looks for the file from the working directory. When deployed through the REST API, it looks for the form field with the same name in the multipart upload.

The following example shows the Spring Integration customStream.xml file.

{ "name" : "springIntegrationStream", "using" : ["defaultFirehose"], "flow" : [ { "springFlow" : { "file": "./examples/streaming-api/springStreamFlow/customStream.xml" } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/springStreamFlow" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }

Example stream


Chapter 4: Examples

Sample data Use the sampleData.json file for this example.

Output event The stream passes the event to the Spring Flow component, which is a bridge that passes the event untransformed back to the stream which then sends the event to the file and AMQP endpoints. The file is written to $RTI_HOME/var/data/distgrid-one/springStreamFlow.

Using Third Party Code Use Spring Integration to inject custom transformers and enrichers into the flow. The JAR files that contain these classes must be in the rti-t/ext/lib directory for every node in the cluster.

The following example shows a Spring configuration that uses a custom transformer:

<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p" xmlns:context="http://www.springframework.org/schema/context" xmlns:c="http://www.springframework.org/schema/c" xmlns:int="http://www.springframework.org/schema/integration" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-4.0.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd">  <int:channel id="input"/> <int:publish-subscribe-channel id="output"/>  <int:bridge input-channel="input" output-channel="output" /> </beans>

12e75770-9bf6-11e6-02ad-8660a71acdf2,234158013224667,447795470718,,359254061234567,1477539186279,1425556331,0,0_0,178 _20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realImsi =;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[]


Chapter 4: Examples

Spring Integration endpoint You can use a custom Spring Integration flow as an endpoint to a stream. The Spring output specification must refer to a spring configuration file that has a channel named input. You can then use any Spring Integration components to manipulate the data and send it out of RTI.

Example endpoints include:

• RTI platform (GemFire)

• MongoDB

• Databases via JDBC

Possible use cases include:

• Stateful enrichment from independent streams.

• Dynamic reference data loading.

• Reduced system down times for new feeds.

The payload sent to the endpoint Spring Integration flow can change based on the endpoint configuration. If the selector expression is *, the payload remains a ProtocolEvent and is sent to the endpoint flow. If the selector expression specifies attributes to select, then the payload is an EventViewObject that contains those attributes. If a formatter is specified for the Spring endpoint, then the payload is a string that is formatted in the configured style.

The RTI Docker image includes the files that are required to use GemFire as the Spring Integration endpoint and details are described below.

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd"> <int:channel id="input" /> <int:channel id="output" /> <int:transformer input-channel="input" output-channel="output" ref="customTransformer" method="transform" /> <bean id="customTransformer" class="com.example.CustomTransformer" /> </beans>

Endpoint payload


Chapter 4: Examples

The following example shows the Spring endpoint specification.

The following example provides a template to define a custom store. For more information, refer to the Spring Integration documentation.

To extend the behavior and flexibility of the system, add GemFire regions dynamically to the platform. Events added to the region can then be used across all the defined streams.

The following shows the example stream $RTI_HOME/examples/streaming-api/springIntegrationEndpoint/customGemfireEndpoint.json file. This spring endpoint puts the ProtocolEvent into a region using the primaryId as the key. The defining Spring file contains the required storage configuration and integration logic to perform the movement of data to end store. This specification refers to the custom-gemfire-endpoint-config.xml file to configure the GemFire endpoint details.

{ "name" : "customOutputStream", "using" : ["defaultFirehose"], "endpoint" : { "selector" : { "expression" : "primaryId" }, "outputs" : [ { "spring": { "formatter": { "type": "CSV" }, "file" : "./my_spring_integration_endpoint.xml" } } ] }, "enabled" : true }

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd"> <int:channel id="input" /> <int:outbound-channel-adapter channel="input" ref="messageStore" method="store"/>  <bean id="messageStore" class="com.example.CustomMessageStore"/> </beans>

Example Spring endpoint specification

Example custom store

GemFire region storage example


Chapter 4: Examples

The following custom-gemfire-endpoint-config.xml example dynamically creates a replicated cluster-wide storage region at runtime and puts received events in to the store using the primaryId key.


{ "name" : "customGemfireEndpoint", "using" : ["defaultFirehose"], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "spring": { "file" : "./examples/streaming-api/springIntegrationEndpoint/custom-gemfire-endpoint-config.xml" } } ] } }

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration" xmlns:gfe="http://www.springframework.org/schema/gemfire" xmlns:int-gfe="http://www.springframework.org/schema/integration/gemfire" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd http://www.springframework.org/schema/gemfire http://www.springframework.org/schema/gemfire/spring-gemfire.xsd http://www.springframework.org/schema/integration/gemfire http://www.springframework.org/schema/integration/gemfire/spring-integration-gemfire.xsd"> <int:channel id="input" />  <gfe:replicated-region id="customStoreRegion"> </gfe:replicated-region> <int-gfe:outbound-channel-adapter id="input" region="customStoreRegion"> <int-gfe:cache-entries> <entry key="payload.primaryId" value="payload" /> </int-gfe:cache-entries> </int-gfe:outbound-channel-adapter> </beans>

Spring data GemFire configuration


Chapter 4: Examples

Output event The output event is inserted into the GemFire region. Use the following commands to launch the GFSH utility that comes with RTI and query the customStoreRegion configured in the Spring Integration file.

[rti@your_host opt]$ cd $RTI_HOME/tools [rti@ your_host tools]$ ls gfsh hdfs-introspector influxdb syphon [rti@ your_host tools]$ cd gfsh/bin [rti@799a41687af1 bin]$ ls gfsh [rti@799a41687af1 bin]$ ./gfsh _________________________ __ / _____/ ______/ ______/ /____/ / / / __/ /___ /_____ / _____ / / /__/ / ____/ _____/ / / / / /______/_/ /______/_/ /_/ v8.2.0 Monitor and Manage GemFire gfsh>connect --locator=localhost[10335] Connecting to Locator at [host=localhost, port=10335] .. Connecting to Manager at [host=799a41687af1, port=1099] .. Successfully connected to: [host=799a41687af1, port=1099] Cluster-2 gfsh>query --query="SELECT * FROM /customStoreRegion" Result : true startCount : 0 endCount : 20 Rows : 1 protectAccess | primaryId | serviceId | sourceId | ingestTime | eventStartTime | eventEndTime | initialLocationId | currentLocationId | protocolName | protocolType | tag | protocolAttributeMap | protocolDetails | uuid | key ------------- | --------------- | ------------ | --------------- | ------------- | -------------- | ------------ | ----------------- | ----------------- | ------------ | ------------ | --- | -------------------- | --------------- | ------------------------------------ | ------------------------------------ false | 234158013224667 | 447795470718 | 359254061234567 | 1477540119275 | 1425556331 | 0 | 0_0 | 178_20108 | adr | 0 | 0 | {} | | 0f4f70d0-6770-4aa9-a387-969b69c85d8a | 3f033bb0-9bf8-11e6-02ad-8660a71acdf2 NEXT_STEP_NAME : END Cluster-2 gfsh>


Chapter 4: Examples

Multiple endpoints Streams can send events to multiple endpoints simultaneously. This is especially useful when external systems need to further analyze the RTI output and derive insight in concert. Endpoints can be configured for different formats and transports.

The following shows the example stream $RTI_HOME/examples/streaming-api/multipleEndpoints/multipleEndpointsStream.json. The stream can send events in both CSV and JSON formats to both file and AMQP endpoints, that is, in all four endpoints.

Example stream


Chapter 4: Examples

{ "name" : "multipleEndpointsStream", "using" : ["defaultFirehose"], "flow" : [ { "enricher": { "compositeKey" : { "attributes" : ["lac", "cellId"], "separator" : "_" }, "enrichAttributes" : "* 'cellTower'", "storeName": "REF_CellTowers" } } ] , "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "JSON" }, "fileBasePath": "${rti.data}/multipleEndpointsJSON" } }, { "amqp": { "formatter": { "type": "JSON" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ENRICH_ALL.JSON", "batchSize": 250, "batchTimeout": 100 } },


Chapter 4: Examples


Output event The output events fall in four endpoints. The CSV file is created in $RTI_HOME/var/data/distgrid-one/multipleEndpointsCSV.

The same event can also be seen in the RabbitMQ queue with a routing key that is bound to the exchange as specified in the multipleEndpointsStream.json file.

{ "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/multipleEndpointsCSV" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ENRICH_ALL.CSV", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }

97dab3c0-a77a-11e6-0d72-0266787ed067,234158013224667,447795470718,,359254061234567,1478805565692,1425556331,0,0_0,178 _20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realImsi =;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[cellTower={20108,G730713,381642.0,218082.0,GL15 5AF,51.8515881255,-2.24084272084,383413.0,217076.0,2500.0,178,51.8515881255,-2.24084272084,NA,2500.0,2500.0,0,0};]


Chapter 4: Examples

The following example shows that the same event is also output in JSON format to a file in $RTI_HOME/var/data/distgrid-one/multipleEndpointsJSON:


Chapter 4: Examples

The following shows that the event appears in JSON format in the RabbitMQ queue, also.

Event enrichment You can enrich events by using colocated reference data.

The following shows that the example stream $RTI_HOME/examples/streaming-api/eventEnricher/eventMobileEquipmentEnricherStream.json uses a single primary key to query reference data and enrich the event if a match is found.

{"uuid":"ed502aa0-c84e-4827-85df-2df34c5bc8d7","key":"97dab3c0-a77a-11e6-0d72-0266787ed067","primaryId":"234158013224 667","serviceId":"447795470718","sourceId":"359254061234567","ingestTime":1478805565692,"eventStartTime":1425556331," eventEndTime":0,"initialLocationId":"0_0","currentLocationId":"178_20108","protocolName":"adr","protocolType":0,"tag" :0,"protocolAttributeMap":{"cellTower":{"cellID":20108,"cellName":"G730713","sitePostCode":"GL15 5AF","siteEasting":3 83413.0,"siteNorthing":217076.0,"siteLatitude":51.8515881255,"siteLongitude":-2.24084272084,"centroidEasting":381642. 0,"centroidNorthing":218082.0,"centroidLatitude":51.8515881255,"centroidLongitude":-2.24084272084,"radii90":2500.0,"r adii80":2500.0,"radii70":2500.0,"cellType":"NA","upstreamBandwidth":0,"downstreamBandwidth":0,"key":"178_20108","lac" :178}},"protocolDetails":{"cellId":20108,"extTags":[],"firstCellId":0,"firstLac":0,"lac":178,"latitude":90.0,"longitu de":180.0,"mccmnc":"23415","radii90":-1.0,"realImei":"","realImsi":"","realMsisdn":"","realTarget":"","status":0},"pr otectAccess":false}

Primary key example


Chapter 4: Examples

Reference data The following example shows device reference data, which is in $RTI_HOME/examples/streaming-api/referenceData/device.csv, that is loaded into RTI.

{ "name" : "eventMobileEquipmentEnricherStream", "using" : ["defaultFirehose"], "flow" : [ { "enricher" : { "primaryKey" : "tac", "enrichAttributes" : "* 'mobileEquipment'", "storeName": "REF_MobileEquipment" } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/eventEnricher" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ENRICH_ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }

35925406,Apple,Apple Iphone,3G,APPLE_DEVICE_PROFILE 01159800,Samsung,Samsung 2760,4G,SAMSUNG_DEVICE_PROFILE


Chapter 4: Examples

Sample data The following example shows the sample data for this stream, which is in the eventMobileEquipmentEnricherStreamSampleData.json file in the sampleData directory. The event includes a tac field that matches the reference data to enrich the event with the device details.

Output event The following shows the output event, which is enriched with the complete device details record and is written to the $RTI_HOME/var/data/distgird-one/eventEnricher directory, as well as to the AMQP endpoint.

The following shows the eventCellTowerAttributesEnricherStream.json file, which is in the eventEnricher directory, and demonstrates the use of the composite key to query reference data and enrich if a match is found. Only certain attributes from the reference data record are used to enrich the event.

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"35925406123456789","eventStartTime":1425556331,"protocolDetails":{"lac":"178","cellId":"20108","tac":"35925406"}}

a7e33410-9bc0-11e6-02ad-8660a71acdf2,234158013224667,447795470718,,35925406123456789,1477516243408,1425556331,0,0_0,1 78_20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;tac=35925406;lac=178;realMsisdn=;radii90=-1.0;cellId =20108;realImsi=;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[mobileEquipment={35925406,Apple,Apple Iphone,3 G,APPLE_DEVICE_PROFILE};]

Composite Key example


Chapter 4: Examples

Reference data The following example shows cell tower reference data, which is in $RTI_HOME/examples/streaming-api/referenceData/cell_tower.csv, that is loaded in RTI.

{ "name" : "eventCellTowerAttributesEnricherStream", "using" : ["defaultFirehose"], "flow" : [ { "enricher": { "compositeKey" : { "attributes" : ["lac", "cellId"], "separator" : "_" }, "enrichAttributes" : "cellID,CellLAC,siteLatitude,siteLongitude", "storeName": "REF_CellTowers" } } ] , "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/eventEnricher" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ENRICH_ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }


Chapter 4: Examples

Sample data Use the sampleData.json file in the sampleData folder for this example. The event includes the composite key values that match the reference data.

Output event The following shows the output event, which is enriched with certain attributes from the matching cell tower record and is written to the $RTI_HOME/var/data/distgrid-one/eventEnricher directory, as well as the AMQP endpoint.

Telco event enrichment Use transformers and enrichers to enrich the protocol event with telco-related attributes.

RTI provides enables you to extract information from IMSI and IMEISV and enriches the attribute to protocol event. The following shows example stream $RTI_HOME/examples/streaming-api/telcoEventEnricher/telcoAttributesEnrichedEventStream.json. It uses rtiTelcoSupportMethods.groovy to enrich the protocolDetails map with following attributes:

• mcc: Mobile Country Code

• mnc: Mobile Network Code

• mccmnc: MCC and MNC,

• tac: Type Allocation Code

• svn: IMEISV Software Version Number

For information about the scripting transformer, refer to Component libraries.



f5b4b5e0-9ac3-11e6-02ad-8660a71acdf2,234158013224667,447795470718,,359254061234567,1477407711294,1425556331,0,0_0,178 _20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realImsi =;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[siteLatitude=51.8515881255;cellID=20108;siteLong itude=-2.24084272084;CellLAC=178;]

Telco supporting attribute enrichment


Chapter 4: Examples

The following example shows the rtiTelcoSupportMethods.groovy file, which extracts information from IMSI and IMEI, and enriches the attributes to protocolDetails map.

{ "name" : "telcoAttributesEnrichedEventStream", "using" : ["defaultFirehose"], "flow" : [ /* Extract required Telco attributes for further processing */ { "scripting": { "file": "./examples/streaming-api/telcoEventEnricher/rtiTelcoSupportMethods.groovy", "method" : "populateAllRequiredTelcoAttributes" } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/telcoEventEnricher" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "TELCO.EVENT.ENRICHER", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }


Chapter 4: Examples

import com.emc.rti.util.imsi.IMSIUtil import io.pivotal.rti.protocols.ProtocolEvent import io.pivotal.rti.t.data.util.IMEIUtil; /** * Provide all required Telco attributes in a single method. * * @param event * @return */ def populateAllRequiredTelcoAttributes(ProtocolEvent event){ setTac( event) setSvn( event) setMnc( event) setMcc( event) setMccMnc( event) } /** * Add Tac to event * * @param event * @return */ def setTac(ProtocolEvent event){ String tac = IMEIUtil.extractTac( event.getSourceId()); if( tac != null){ event.getProtocolDetails().put("tac", tac); } } /** * Add SVN to event * * @param event * @return */ def setSvn(ProtocolEvent event){ String svn = IMEIUtil.extractSvn( event.getSourceId()); if( svn != null ){ event.getProtocolDetails().put("svn", svn); } } /** * Add Mcc to event * * @param event * @return */ def setMcc(ProtocolEvent event){ String mcc = IMSIUtil.extractMcc(event.getPrimaryId()); if( mcc != null ){ event.getProtocolDetails().put("mcc", mcc); } }


Chapter 4: Examples

Sample data The following example shows the sample data for this stream, which is in telcoAttributesEnrichedEventStreamSampleData.json file in the sampleData directory. The event includes various attributes from which new attributes are extracted and enriched by rtiTelcoSupportMethods.groovy.

Output event The following shows the output event, which is enriched and written to the $RTI_HOME/var/data/distgrid-one/telcoEventEnricher directory, as well as the AMQP endpoint.

RTI uses a reference data enricher to enrich a protocol event with mobile equipment data. For more information about mobile equipment reference data and reference data enricher, refer to the EMC Real-Time Intelligence (RTI) Administrators Guide.

/** * Add Mnc to event * * @param event * @return */ def setMnc(ProtocolEvent event){ String mnc = IMSIUtil.extractMnc(event.getPrimaryId()); if( mnc != null ){ event.getProtocolDetails().put("mnc", mnc); } } /** * Add Mnc to event * * @param event * @return */ def setMccMnc(ProtocolEvent event){ String mccmnc = IMSIUtil.extractMccMnc(event.getPrimaryId()); if( mccmnc != null ){ event.getProtocolDetails().put("mccmnc", mccmnc); } }

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"35925406123456789","eventStartTime":1425556331,"protocolDetails":{"lac":"178","cellId":"20108"}}

92266ae0-9add-11e6-02ad-8660a71acdf2,234158013224667,447795470718,,35925406123456789,1477418711182,1425556331,0,0_0,1 78_20108,adr,0,0,,[extTags=[];mcc=234;status=0;firstCellId=0;tac=35925406;realTarget=;svn=78;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realImsi=;longitude=180.0;mnc=15;mccmnc=23415;latitude=90.0;firstLac=0;realImei=;],[]

Mobile equipment enrichment


Chapter 4: Examples

The following shows that the example stream telcoAttributesEnrichedEventStreamMobileEquipment.json, which is in the telcoEventEnricher directory, uses the tac value as the primary key to query the REF_MobileEquipment Gemfire region. The tac attribute must be in the protocolDetails map. If a match is found, the event is enriched with a mobileEquipment attribute, whose value is the matching MobileEquipment object. The mobileEquipment is added to the protocolAttributeMap of the event.

Reference data The following example shows device reference data, which is in $RTI_HOME/examples/streaming-api/referenceData/device.csv, that is loaded into RTI.

{ "name" : "telcoAttributesEnrichedEventStream.mobileEquipment", "using" : ["defaultFirehose"], "flow" : [ { "enricher" : { "primaryKey" : "tac", "enrichAttributes" : "* 'mobileEquipment'", "storeName": "REF_MobileEquipment" } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/telcoEventEnricher" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "TELCO.EVENT.ENRICHER", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }

Example stream


Chapter 4: Examples

Sample data Use the telcoAttributesEnrichedEventStreamMobileEquipmentSampleData.json file in the sampleData directory. The event includes a tac field that matches the reference data to enrich the event with the device details.

Output event The following shows the output event, which is enriched with the complete device details record and is written to the $RTI_HOME/ var/data/distgrid-one/telcoEventEnricher directory, as well as to the AMQP endpoint.

RTI uses a reference data enricher to enrich a protocol event with cell tower data. For more information about cell tower reference data and reference data enricher, refer to the EMC Real-Time Intelligence (RTI) Administrators Guide.

The following shows that the example stream telcoAttributesEnrichedEventStreamCellTower.json, which is in the telcoEventEnricher directory, uses the value of lac, for example 123, and the value of cellId, for example 456, to construct a key by using "_" as separator, resulting in the composite key 123_456. The lac and cellId attributes must be in the event’s protocolDetails map. The enricher uses this key to query the REF_CellTowers Gemfire region. If a match is found, the the event is enriched with a cellTower attribute whose value is the matching cell tower object. The cellTower attribute is added to the protocolAttributeMap of the event.

35925406,Apple,Apple Iphone,3G,APPLE_DEVICE_PROFILE 01159800,Samsung,Samsung 2760,4G,SAMSUNG_DEVICE_PROFILE

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"35925406123456789","eventStartTime":1425556331,"protocolDetails":{"lac":"178","cellId":"20108","tac":"35925406"}}

10df5de0-9b03-11e6-02ad-8660a71acdf2,234158013224667,447795470718,,35925406123456789,1477434815165,1425556331,0,0_0,1 78_20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;tac=35925406;lac=178;realMsisdn=;radii90=-1.0;cellId =20108;realImsi=;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[mobileEquipment={35925406,Apple,Apple Iphone,3 G,APPLE_DEVICE_PROFILE};]

Cell tower enrichment


Chapter 4: Examples

Reference data The following example shows cell tower reference data, which is in $RTI_HOME/examples/streaming-api/referenceData/cell_tower.csv, which is loaded into RTI.

{ "name" : "telcoAttributesEnrichedEventStream.cellTower", "using" : ["defaultFirehose"], "flow" : [ { "enricher" : { "compositeKey" : { "attributes" : ["lac", "cellId"], "separator" : "_" }, "enrichAttributes" : "* 'cellTower'", "storeName": "REF_CellTowers" } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/telcoEventEnricher" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "TELCO.EVENT.ENRICHER", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }


Chapter 4: Examples

Sample data Use the sampleData.json file in the sampleData. The event has the composite key values that match the reference data.

Output event The following shows the output event, which is enriched with the complete matching cell tower record and is written to the $RTI_HOME/ var/data/distgrid-one/telcoEventEnricher directory, as well as the AMQP endpoint.

RTI uses a reference data enricher to enrich a protocol event with with subscriber profile data. For more information about subscriber profile reference data and reference data enricher, refer to the EMC Real-Time Intelligence (RTI) Administrators Guide.

The following shows that the example stream telcoAttributesEnrichedEventStreamSubscriberProfile.json, which is in the telcoEventEnricher directory, uses the imsi value to query the REF_SubscriberProfiles Gemfire region. The imsi attribute must be in protocolDetails map. If a match is found, the event is enriched with a subscriberProfile attribute, whose value is the matching subscriber profile record. The subscriberProfile attribute is added to the protocolAttributeMap.



d043c140-9b02-11e6-02ad-8660a71acdf2,234158013224667,447795470718,,359254061234567,1477434706772,1425556331,0,0_0,178 _20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realImsi =;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[cellTower={20108,G730713,381642.0,218082.0,GL15 5AF,51.8515881255,-2.24084272084,383413.0,217076.0,2500.0,178,51.8515881255,-2.24084272084,NA,2500.0,2500.0,0,0};]

Subscriber profile enrichment


Chapter 4: Examples

Reference data The following example shows SubscriberProfile reference data, which is in $RTI_HOME/examples/streaming-api/referenceData/subscriber_profile.csv, which is loaded into RTI.

Sample data Use the telcoAttributesEnrichedEventStreamSubscriberProfileSampleData.json file in the sampleData directory. The event includes the imsi key value that matches the reference data.

{ "name" : "telcoAttributesEnrichedEventStream.subscriberProfile", "using" : ["defaultFirehose"], "flow" : [ { "enricher" : { "primaryKey" : "imsi", "enrichAttributes" : "* 'subscriberProfile'", "storeName": "REF_SubscriberProfiles" } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/telcoEventEnricher" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "TELCO.EVENT.ENRICHER", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }

234158013224667,447795470718,90210,dell;emc;vip,externalId1 564537281964889,564278346195,12345,red;blue;gold,externalId2


Chapter 4: Examples

Output event The following shows the output event, which is enriched with the complete matching subscriber profile record and is written to the $RTI_HOME/var/data/distgrid-one/telcoEventEnricher directory, as well as the AMQP endpoint.

The following shows the example stream telcoAttributesEnrichedEventStreamSubscriberProfileMsisdn.json, which is in the telcoEventEnricher directory. It is similar to the example stream telcoAttributesEnrichedEventStreamSubscriberProfile.json, except that the msisdn attribute from the matching subscriber profile is used to enrich the protocol event.

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"35925406123456789","eventStartTime":1425556331,"protocolDetails":{"imsi":"234158013224667","lac":"178","cellId":"20108"}}

70292290-9b03-11e6-02ad-8660a71acdf2,234158013224667,447795470718,,35925406123456789,1477434975033,1425556331,0,0_0,1 78_20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realIm si=;longitude=180.0;imsi=234158013224667;latitude=90.0;firstLac=0;realImei=;],[subscriberProfile={234158013224667,447 795470718,90210,[dell, vip, emc],externalId1};]


Chapter 4: Examples


Sample data Use the telcoAttributesEnrichedEventStreamSubscriberProfileSampleData.json file in the sampleData directory. The event includes the imsi key value that matches the reference data.

{ "name" : "telcoAttributesEnrichedEventStream.subscriberProfileMsisdn", "using" : ["defaultFirehose"], "flow" : [ { "enricher" : { "primaryKey" : "imsi", "enrichAttributes" : "msisdn", "storeName": "REF_SubscriberProfiles" } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/telcoEventEnricher" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "TELCO.EVENT.ENRICHER", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }



Chapter 4: Examples

Output event The following shows the output event, which is enriched with only the msisdn attribute from the matching subscriber profile record and is written to the $RTI_HOME/var/data/distgrid-one/telcoEventEnricher directory, as well as the AMQP endpoint.

Number attribute enrichment This example enriches the event with new attributes, which have values that are the same as certain attributes found in the protocolDetails map. These new attributes are added to the ProtocolAttributes map. The stream specification lists the attributes from the protocolDetails map and the corresponding new enriched attribute names. Use this form of enrichment when other components, such as metrics calculators, use the enriched attributes downstream.

The following shows that the example file $RTI_HOME/examples/streaming-api/numberAttributeEnricher/numberAttributeEnricher adds uploadThroughput and downloadThroughput to protocolAttributeMap. The value of uploadThroughput is from bytesUplink and the value of downloadThroughput is from bytesDownlink.

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"35925406123456789","eventStartTime":1425556331,"protocolDetails":{"imsi":"234158013224667","lac":"178","cellId":"20108"}}

fde0b3e0-9b8b-11e6-02ad-8660a71acdf2,234158013224667,447795470718,,35925406123456789,1477493624350,1425556331,0,0_0,1 78_20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realIm si=;longitude=180.0;imsi=234158013224667;latitude=90.0;firstLac=0;realImei=;],[msisdn=447795470718;]

Example stream


Chapter 4: Examples

Sample data

Use the numberAttributeEnricherSampleData.json file in the sampleData directory for this example.

Output event The following shows the output event, which is enriched as per the specification and is written to the $RTI_HOME/var/data/distgrid-one/numberAttributeEnricher directory, as well as the AMQP endpoint.

{ "name" : "numberAttributeEnricher", "using" : ["defaultFirehose"], "enabled" : true, "flow" : [ { "numberAttributeEnricher": { "numberAttributeEnrichment" : [ { "attribute" : "bytesUplink", "enrichAttribute" : "uploadThroughput", "type" : "long" }, { "attribute" : "bytesDownlink", "enrichAttribute" : "downloadThroughput", "type" : "long" } ] } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/numberAttributeEnricher" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "NumberAttributeEnricher.STREAM", "routingKey": "NumberAttributeEnricher.STREAM", "batchSize": 1000, "batchTimeout": 250 } } ] }}

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"359254061234567","eventStartTime":1425556331,"protocolDetails":{"mccmnc":"23415","lac":"178","cellId":"20108","bytesUplink":1000,"bytesDownlink":1000}}


Chapter 4: Examples

Number attribute enrichment also supports addition operation. The following shows that the example stream numberAttributeEnricherTotal.json enriches the protocolAttributeMap with the totalThroughput attribute whose value is the sum of bytesUplink and bytesDownlink.

8f7424c0-9a30-11e6-6c10-15e57678b3d7,234158013224667,447795470718,,359254061234567,1477344403724,1425556331,0,0_0,178 _20108,adr,0,0,,[bytesDownlink=1000;extTags=[];status=0;firstCellId=0;bytesUplink=1000;realTarget=;lac=178;realMsisdn =;radii90=-1.0;cellId=20108;realImsi=;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[uploadThroug hput=1000;downloadThroughput=1000;]

{ "name" : "numberAttributeEnricherTotal", "using" : ["defaultFirehose"], "enabled" : true, "flow" : [ { "numberAttributeEnricher": { "numberAttributeEnrichment" : [ { "attribute" : "bytesUplink + bytesDownlink", "enrichAttribute" : "totalThroughput", "type" : "long" } ] } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/numberAttributeEnricher" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "NumberAttributeEnricher.STREAM", "routingKey": "NumberAttributeEnricher.STREAM", "batchSize": 1000, "batchTimeout": 250 } } ] } }


Chapter 4: Examples

Sample data

Use the numberAttributeEnricherSampleData.json file in the sampleData directory for this example.

Output event The following shows the output event, which is enriched according to the specification with the summed value and is written to the $RTI_HOME/var/data/distgrid-one/numberAttributeEnricher directory, as well as the AMQP endpoint.

Condition enrichment The following shows that the example stream $RTI_HOME/examples/streaming-api/conditionEnricher/conditionEnricher.json file, adds flags according to a logical SpEL-based expression that evaluates the incoming event. The stream adds the errorCountAboveThreshold attribute with value true to the protocolAttributeMap if the errorCount attribute in the protocolDetails map is greater than 3.

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"359254061234567","eventStartTime":1425556331,"protocolDetails":{"mccmnc":"23415","lac":"178","cellId":"20108","bytesUplink":1000,"bytesDownlink":1000}}

48ccd760-9a3f-11e6-6c10-15e57678b3d7,234158013224667,447795470718,,359254061234567,1477350727638,1425556331,0,0_0,178_20108,adr,0,0,,[bytesDownlink=1000;extTags=[];status=0;firstCellId=0;bytesUplink=1000;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realImsi=;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[totalThroughput=2000;]

Example stream


Chapter 4: Examples

Sample data Use the conditionEnricherSampleData.json file in the sampleData directory for this example. The event has the errorCount attribute with a value of 10.

{ "name" : "conditionEnricher", "using" : ["defaultFirehose"], "enabled" : true, "flow" : [ { "conditionEnricher": { "conditionEnrichment" : [ { "condition" : "protocolDetails.containsKey('errorCount') && protocolDetails['errorCount'] > 3", "enrichAttribute": "errorCountAboveThreshold" } ] } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/conditionEnricher" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "ConditionEnricher.STREAM", "routingKey": "ConditionEnricher.STREAM", "batchSize": 1000, "batchTimeout": 250 } } ] }}

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"359254061234567","eventStartTime":1425556331,"protocolDetails":{"mccmnc":"23415","lac":"178","cellId":"20108","errorCount":10}}


Chapter 4: Examples

Output event The following shows the output event, which is enriched according to the specification because the value of errorCount in the input event exceeds the threshold of 3. The event is written to the $RTI_HOME/var/data/distgrid-one/conditionEnricher directory, as well as to the AMQP endpoint.

Component SDK Build custom components by using the Streaming API Component SDK to process events in different ways. For more information about the examples included in the RTI Docker image, refer to Component SDK.

The following examples are in the $RTI_HOME/examples/streaming-api/componentSDK directory. The Java custom components are in the componentSDK/src/main/java directory tree and the stream examples that use these components are in the componentSDK/src/main/resources directory. There is also a componentSDK/src/main/test directory with tests for the custom Java components that you can modify and test.

A build.gradle file is also provided with all the dependencies to build the components. To build, run the gradle distTarGz command to create the component-sdk-examples-1.0.tar.gz file in the componentSDK/build/distributions/ directory. Use the RTISH rti ext install command to deploy this file to an RTI cluster and then restart the service grid instance.

Note: So that you can run the stream examples, the file has already been built and deployed under $RTI_HOME/ext/lib.

For information about RTISH commands, refer to the EMC Real-Time Intelligence (RTI) Administration Guide.

The following shows the example stream $RTI_HOME/examples/streaming-api/componentSDK/src/main/resources/callTypeFilterStream.json. This stream uses the callTypeFilter component to filter out events that do not have a callType of 9.

6e406fd0-9a1b-11e6-6c10-15e57678b3d7,234158013224667,447795470718,,359254061234567,1477335328589,1425556331,0,0_0,178 _20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;errorCount=10;lac=178;realMsisdn=;radii90=-1.0;cellId= 20108;realImsi=;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[errorCountAboveThreshold=true;]

CallTypeFilterStream example


Chapter 4: Examples

Sample data Use the callTypeFilterStreamSampleData.json file in the sampleData directory for this example.

{ "name": "callTypeFilterStream", "using": ["defaultFirehose"], "enabled": true, "flow": [ { "callTypeFilter": { // 9 = SMS "callType": 9 } } ], "endpoint": { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "batchSize": 8192, "batchTimeout": 15000, "fileBasePath": "${rti.data}/componentSDK" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] } }

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"359254061234567","eventStartTime":1425556331,"protocolDetails":{"mccmnc":"23415","lac":"178","cellId":"20108","callType":"3"}} {"protocolName":"adr","primaryId":"55555555","serviceId":"6666666","sourceId":"7754526840","eventStartTime":1425573968,"protocolDetails":{"mccmnc":"23415","lac":"178","cellId":"20108","callType":"9"}}


Chapter 4: Examples

Output event The following shows the output event. Only the event with callType=9 is allowed to pass. Output is written to the $RTI_HOME/var/data/distgrid-one/componentSDK directory, as well as to the AMQP endpoint.

The following shows that the destinationIPObfuscatorStream.json file in the src/main/resources directory changes the destination IP value to either the routableIP or nonROutableIP value specified, depending on the type of the destination IP in the incoming event.

8ca84400-9d24-11e6-3422-33493239903f,55555555,6666666,,7754526840,1477669098560,1425573968,0,0_0,178_20108,adr,0,0,,[ extTags=[];status=0;firstCellId=0;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realImsi=;mccmnc=23415;lo ngitude=180.0;latitude=90.0;firstLac=0;realImei=;callType=9;],[]

DestinationIPObfuscatorStream example


Chapter 4: Examples

Sample data Use the destinationIPObfuscatorStreamSampleData.json file in the sampleData directory for this example.

{ "name": "destinationIPObfuscatorStream", "using": ["defaultFirehose"], "enabled": true, "flow": [ { "destinationIPObfuscator": { "routableIP": "129.97.140.10", "nonRoutableIP": "10.1.0.1" } } ], "endpoint": { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "batchSize": 8192, "batchTimeout": 15000, "fileBasePath": "${rti.data}/componentSDK" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] } }

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"359254061234567","eventStartTime":1425556331,"protocolDetails":{"mccmnc":"23415","lac":"178","cellId":"20108","destinationIp":"192.168.99.100"}} {"protocolName":"adr","primaryId":"55555555","serviceId":"6666666","sourceId":"7754526840","eventStartTime":1425573968,"protocolDetails":{"mccmnc":"23415","lac":"178","cellId":"20108","destinationIp":"151.101.20.73"}}


Chapter 4: Examples

Output event The following shows the output event. The events were enriched based on the destinationIP type. Output is written to the $RTI_HOME/var/data/distgrid-one/componentSDK directory, as well as to the AMQP endpoint.

GroupMembershipEnricherStream example The following shows that the example stream groupMembershipEnricherStream.json in the src/main/resources directory employs the GroupMembershipEnricher component to enrich an event with the groups of the subscriber profile associated with the event’s primaryId. The enricher queries Subscriber Profile reference data to find a record whose group information is added to the event.

aae2c5e0-9d28-11e6-3422-33493239903f,234158013224667,447795470718,,359254061234567,1477670867261,1425556331,0,0_0,178 _20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;destinationIp=10.1.0.1;lac=178;realMsisdn=;radii90=-1. 0;cellId=20108;realImsi=;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[] aae2ecf0-9d28-11e6-3422-33493239903f,55555555,6666666,,7754526840,1477670867263,1425573968,0,0_0,178_20108,adr,0,0,,[ extTags=[];status=0;firstCellId=0;realTarget=;destinationIp=129.97.140.10;lac=178;realMsisdn=;radii90=-1.0;cellId=201 08;realImsi=;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[]


Chapter 4: Examples


{ "name": "groupMembershipEnricherStream", "using": ["defaultFirehose"], "enabled": true, "flow": [ { "groupMembershipEnricher": { "storeName": "REF_SubscriberProfiles" } } ], "endpoint": { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "batchSize": 8192, "batchTimeout": 15000, "fileBasePath": "${rti.data}/componentSDK" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ALL", "batchSize": 250, "batchTimeout": 100 } } ] } }



Chapter 4: Examples

Sample data Use the groupMembershipEnricherStreamSampleData.json file in the sampleData directory for this example. The event has the primaryId key value that matches the reference data.

Output event The following shows the output event, which is enriched with the groups from the matching subscriber profile record and is written to the $RTI_HOME/var/data/distgrid-one/componentSDK director, as well as to the AMQP endpoint.

Kafka output The following shows that the example stream $RTI_HOME/examples/streaming-api/kafka/allKafkaStreamCreateTopic.json sends events to a Kafka endpoint. The createTopic attribute is true and as a result the DECODER.STREAM.CUSTOM topic is created by RTI before RTI sends events to it.

{"protocolName":"adr","primaryId":"234158013224667","serviceId":"447795470718","sourceId":"35925406123456789","eventStartTime":1425556331,"protocolDetails":{"lac":"178","cellId":"20108"}}

0e735e50-a574-11e6-a6bc-d9fc1399f492,234158013224667,447795470718,,35925406123456789,1478582855861,1425556331,0,0_0,1 78_20108,adr,0,0,,[radii90=-1.0;extTags=[];cellId=20108;realImsi=;status=0;firstCellId=0;realTarget=;longitude=180.0; latitude=90.0;firstLac=0;realImei=;lac=178;realMsisdn=;],[profile.groups=dell+vip+emc;]


Chapter 4: Examples

The following shows the example stream allKafkaStream.json file, which is similar to the previous example, except that the createTopic attributed and other related required attributes are not set. By default the topic is created.

{ "name" : "allKafkaStreamCreateTopic", "using" : ["defaultFirehose"], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "kafka": { "formatter": { "type": "CSV" }, "bootstrapServers": ["127.0.0.1:9092"], "topic": "DECODER.STREAM.CUSTOM", "clientId": "demoRTI", "createTopic": true, "zookeeperServers": ["127.0.0.1:2181"] } } ] }, "enabled" : true }

{ "name" : "allKafkaStream", "using" : ["defaultFirehose"], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "kafka": { "formatter": { "type": "CSV" }, "bootstrapServers": ["127.0.0.1:9092"], "topic": "DECODER.STREAM.CUSTOM", "clientId": "demoRTI" } } ] }, "enabled" : true }


Chapter 4: Examples

Start Kafka To start both Zookeeper and the Kafka server, use the following script.

Alternatively, you can start these components by running the scripts under the /bin directory in the Kafka installation which is in /opt.

Start a consumer process Change to the Kafka installation directory under which is in /opt. To start a consumer process, use the following script.

Sample data Use the sampleData.json file for this example and send the event to RTI (from another terminal window).

Output event The following shows the output event on the Kafka consumer terminal window after you pass the sample event.

d5a20100-a0f3-11e6-8dda-1c8327a8e619,234158013224667,447795470718,,359254061234567,1478087980299,1425556331,0,0_0,178_20108,adr,0,0,,[extTags=[];status=0;firstCellId=0;realTarget=;lac=178;realMsisdn=;radii90=-1.0;cellId=20108;realImsi=;mccmnc=23415;longitude=180.0;latitude=90.0;firstLac=0;realImei=;],[]

Event Stream Subscription (ESS) Create Event Stream Subscriptions by deploying a stream so that consumers can receive events that match the subscription criteria. The criteria are specified in the stream. The following example shows the $RTI_HOME/examples/streaming-api/ess/essStream.json file.

The essStream uses the LANDSOptInBlacklistStream as its input source. Events from the LANDSOptInBlacklistStream are sent to the eventTimeFilter, which only allows events in the specified time interval to pass on. For more information about the LANDSOptinInBlacklistStream, refer to Location subscriber opt-in streams.

These events pass through the geofenceByCellIdEnricher in which events whose Lac and Cell Id match cell towers in a specified geofence are enriched with the containedGeoFences tag and geofence name (for example, UK). The geofenceFilter allows only those events with the containdGeoFences value set to UK to pass through.

[rti@your_host opt]$ emc_rti_dist/run_kafka/kafka-start.sh

[rti@your_host kafka_2.11-0.9.0.1]$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --zookeeper localhost:2181 --topic DECODER.STREAM.CUSTOM --from-beginning


Chapter 4: Examples

{ "name" : "essStream", "using" : ["LANDSOptInBlacklistStream"], "flow" : [ { "eventTimeFilter" : { "from": "09/15/2016", "to": "09/17/2016" } }, { "geofenceByCellIdEnricher" : { "geofences": [ { "id" : "UK", "geometry" : "POLYGON ((-1 51, -1 52, 0 52, 0 51, -1 51))" } ] } }, { "geofenceFilter" :{ "geofenceIds" : ["UK"] } } ], "endpoint" : { "selector" : { "expression" : "*" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/ess" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "DECODER.STREAM.CUSTOM", "routingKey": "ESS", "batchSize": 250, "batchTimeout": 100, "connectionSetting" : { "username" : "guest", "password" : "guest", "hostAddress" : "localhost:5672" } }


Chapter 4: Examples

Deploy the LANDSOptinInBlacklistStream as described in Location subscriber opt-in streams.

Deploy the essStream.json using either:

• The RTISH rti stream deploy command

• The REST API, if you do not have operator access to the service grid and RTISH. On the image, the REST API server runs on port 7080. For example using the CURL command:

Reference data For information about the location and subscriber Opt-In reference data, refer to Location subscriber opt-in streams. For information about cell tower reference data, refer to Geofence detection.

Sample data Use the essStreamSampleData.json file in the sampleData directory for this example. Note that the event attributes determine how the events are filtered and enriched by the components in the stream.


Chapter 4: Examples

Output event The following shows the output event of the LANDSOptInBlacklistStream. The output events, based on the optin filters and reference data, are written to file in $RTI_HOME/var/data/distgrid-one/optinStreams directory, as well as to the AMQP endpoint specified in that stream.

The eventTimeFilter prevents the event with primaryId 11111 from passing through because its eventEndTime is not in the time window specified in the filter. The two remaining events pass onto the geofenceByCellIdEnricher and the geofenceFilter after which they are sent to the endpoints.

The following shows the output event from essStream , which is written to file in the $RTI_HOME/var/data/distgrid-one/ess directory, as well as to the AMQP endpoint. Only the event with primaryId 66666 is output, because its LAC and Cell ID match the cell tower found in the geofence specified in the geofenceByCellIdEnricher. The event is tagged with the UK geofence and as a result it is allowed to pass through the geofenceFilter. The other event is not enriched and is dropped by the filter.


Chapter 4: Examples

Non-telco streams The Streaming API can be used for more than just Telco applications. Its features can be applied in any domain that needs stream-processing capabilities. For example, the $RTI_HOME/examples/streaming-api/nonTelco/nonTelcoStream.json file shown below consists of a stream that can be used as a simple stock watcher stream. The expressionFilter ensures that only events that have a particular stock symbol and price that is greater than a certain threshold are allowed to pass. For such events, the stream sends a notification over AMQP and writes it to file.

{ "name" : "nonTelcoStream", "using" : ["defaultFirehose"], "flow" : [ { "expressionFilter": { "expression": "primaryId.equals('EMC') && protocolDetails['price'] > 30" } } ], "endpoint" : { "selector" : { "expression" : "primaryId,price" }, "outputs" : [ { "file": { "formatter": { "type": "CSV" }, "fileBasePath": "${rti.data}/nonTelco" } }, { "amqp": { "formatter": { "type": "CSV" }, "exchange": "STOCKS", "routingKey": "STOCKS.PRICE.ALERT", "batchSize": 250, "batchTimeout": 100 } } ] }, "enabled" : true }


Chapter 4: Examples

Sample Data Stock data can be input to RTI using the JSON format. The nonTelcoStreamSampleData.json file listed below is in the sampleData folder:

Output Event The output from the stream is sent to the AMQP endpoint and also written to file under $RTI_HOME/var/data/distgrid-one/nonTelco. Only the event matching the criteria in the expressionFilter passes. The following output consists of attribute values specified in the selector:

EMC,40

KPI stores KPI stores enable you to save historical metrics calculation results. There are two KPI store regions in each distribution grid node:

• The LOCAL store is a partitioned region that is used to save the KPI results with large group-by values.

• The GLOBAL store is a replicated region that is used to save the KPI results with smaller group-by values.

For information about LOCAL and GLOBAL stores, refer to the EMC Real-Time Intelligence (RTI) Administrators Guide. For information about KPI store attributes, refer to the EMC Real-Time Intelligence (RTI) Developer’s Guide.

To use the KPI store function, obtain the emc-rti-kpi-store-version-build_number.tar.gz package from your RTI representative. This file contains libraries required for the KPI store that must be installed in the service grid. For information about installing extension packages, refer to the EMC Real-Time Intelligence (RTI) Administrators Guide.

Note: The RTI Docker image already has the extension package installed under $RTI_HOME/ext/lib.

This example shows how to enrich an event with analytic attributes by using an enrichment stream. It also shows how to subscribe to an enrichment stream and configure a KPI store by using a metrics stream.


Chapter 4: Examples

The following enrichment stream enriches the Affirmed stream with analytic attributes. The example stream affirmedStream.json is in the $RTI_HOME/examples/streaming-api/kpiStore directory.

{ "name" : "affirmedStream", "using" : ["defaultFirehose"], "enabled" : true, "flow" : [ { "protocolFilter": { "type" : "affirmed-edr", "subType": 4 } }, { "expressionFilter": { "expression": "protocolDetails['flowRecordType'] == 2" } }, { "numberAttributeEnricher": { "numberAttributeEnrichment" : [ { "attribute" : "upLinkOctets", "enrichAttribute" : "affirmed.flow.throughput.upload", "type" : "long" }, { "attribute" : "downLinkOctets", "enrichAttribute" : "affirmed.flow.throughput.download", "type" : "long" }, { "attribute" : "upLinkDropOctets", "enrichAttribute" : "affirmed.flow.dropOctets.upload", "type" : "long" }, { "attribute" : "downLinkDropOctets", "enrichAttribute" : "affirmed.flow.dropOctets.download", "type" : "long" }, { "attribute" : "upLinkPackets", "enrichAttribute" : "affirmed.flow.packets.upload", "type" : "long" }, { "attribute" : "downLinkPackets", "enrichAttribute" :


Chapter 4: Examples

"affirmed.flow.packets.download", "type" : "long" }, { "attribute" : "upLinkDropPackets", "enrichAttribute" : "affirmed.flow.dropPackets.upload", "type" : "long" }, { "attribute" : "downLinkDropPackets", "enrichAttribute" : "affirmed.flow.dropPackets.download", "type" : "long" } ] } }, { "protocolEventToSubscriber": { "includeAttributes" : true, "includeDetail" : true } } ] } After the analytic attributes are enriched to the event, the following metrics stream subscribes to the enrichment stream and performs the metrics calculation. After the KPI results are calculated, they are saved to the LOCAL KPI store. The example metrics stream affirmedLocalKpi.json is in the $RTI_HOME/examples/streaming-api/kpiStore directory, as shown in the following:

{ "name" : "affirmedLocalKpi", "using" : ["affirmedStream"], "enabled" : true, "groupBy" : "protocolDetailMap['application']", "groupBySizingHint" : 500, "eventTime" : "INGEST_TIME", "statisticsEnabled" : true, "metrics" : { "includePatterns" : [ "affirmed.flow.**"] }, "export" : { "fields" : [ "affirmed.flow.throughput.upload", "affirmed.flow.throughput.download", "affirmed.flow.throughput.total", "affirmed.flow.packets.upload", "affirmed.flow.packets.download", "affirmed.flow.packets.total", "affirmed.flow.dropPackets.upload", "affirmed.flow.dropPackets.download", "affirmed.flow.dropPackets.total", "affirmed.flow.dropPackets.ratio.upload", "affirmed.flow.dropPackets.ratio.download", "affirmed.flow.dropPackets.ratio.total"


Chapter 4: Examples

] }, "xbar" : { "slidePeriod" : 60000, "windowLength" : 60000 }, "KPIStore" : { "storeType" : "LOCAL", "enabled" : true, "TTL" : 3 } } To deploy the metrics stream, use the rti metrics deploy command:

rtish> rti metrics deploy --definition examples/streaming-api/kpiStore/affirmedLocalKpi.json

For information about command details, refer to the EMC Real-Time Intelligence (RTI) Administrators Guide.

The following shows another similar example, which is the $RTI_HOME/examples/streaming-api/kpiStore/affirmedGlobalKpi.json metrics stream for the GLOBAL KPI store.

{ "name" : "affirmedGlobalKpi", "using" : ["affirmedStream"], "enabled" : true, "groupBy" : "protocolDetailMap['application']", "groupBySizingHint" : 500, "eventTime" : "INGEST_TIME", "statisticsEnabled" : true, "metrics" : { "includePatterns" : [ "affirmed.flow.**"] }, "export" : { "fields" : [ "affirmed.flow.throughput.upload", "affirmed.flow.throughput.download", "affirmed.flow.throughput.total", "affirmed.flow.packets.upload", "affirmed.flow.packets.download", "affirmed.flow.packets.total", "affirmed.flow.dropPackets.upload", "affirmed.flow.dropPackets.download", "affirmed.flow.dropPackets.total", "affirmed.flow.dropPackets.ratio.upload", "affirmed.flow.dropPackets.ratio.download", "affirmed.flow.dropPackets.ratio.total" ] }, "xbar" : { "slidePeriod" : 60000, "windowLength" : 60000 }, "KPIStore" : { "storeType" : "GLOBAL",


Chapter 4: Examples

"enabled" : true, "TTL" : 3 } } To deploy the metrics stream, use the rti metrics deploy command:

rtish> rti metrics deploy --definition examples/streaming-api/kpiStore/affirmedGlobalKpi.json

Sample data Use the $RTI_HOME/examples/streaming-api /sampleData/kpiStoreSampleData.json file for this example.

Result Checking There is no external API to access the KPI store. Check the result by using the gfsh command to query the localKpiStore and globalKpiStore region in RTI. For information about the gfsh command, refer to the GemFire documentation.

In the following example, the gfsh command queries the LOCAL and GLOBAL regions:

[rti@your_host opt]$ cd $RTI_HOME/tools/gfsh/bin/ [rti@your_host bin]$ ./gfsh _________________________ __ / _____/ ______/ ______/ /____/ / / / __/ /___ /_____ / _____ / / /__/ / ____/ _____/ / / / / /______/_/ /______/_/ /_/ v8.2.0 Monitor and Manage GemFire gfsh>connect --locator=localhost[10335] Connecting to Locator at [host=localhost, port=10335] .. Connecting to Manager at [host=e3ca37febc59, port=1099] .. Successfully connected to: [host=e3ca37febc59, port=1099] Cluster-2 gfsh>query --query="select entry.value from /globalKpiStore.entries entry" Result : true startCount : 0 endCount : 20 Rows : 1 kpiResults | metricName | groupByName ------------------------------- | ----------------- | ----------- com.gemstone.org.json.JSONArray | affirmedGlobalKpi | 24


Chapter 4: Examples

NEXT_STEP_NAME : END Cluster-2 gfsh>query --query="select entry.value from /localKpiStore.entries entry" Result : true startCount : 0 endCount : 20 Rows : 2 kpiResults | metricName | groupByName ------------------------------- | ---------------- | ----------- com.gemstone.org.json.JSONArray | affirmedLocalKpi | 24 com.gemstone.org.json.JSONArray | master | master NEXT_STEP_NAME : END

Documents

EMC REAL-TIME INTELLIGENCE (RTI) 3.1 - VMware · EMC REAL-TIME INTELLIGENCE (RTI) 3.1.0 Streaming Guide . April 2017 . Abstract This reference guide describes the RTI streaming platform