Messaging Toolkit InfoSphere Streams Version 3.0

© 2012 IBM Corporation1

Messaging Toolkit

InfoSphere Streams Version 3.0

Anjali AgarwalToolkit


Important Disclaimer

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.

WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.

IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

• CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR

• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE.

The information on the new product is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information on the new product is for informational purposes only and may not be incorporated into any contract. The information on the new product is not a commitment, promise, or legal obligation to deliver any material, code or functionality. The development, release, and timing of any features or functionality described for our products remains at our sole discretion.

THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.


Agenda

Introduction to Messaging Toolkit Use cases -Why use Messaging Toolkit Operators Terminologies and Concepts Software Dependency Enhancements/Features common to XMSSource and XMSSink Enhancements/Features for XMSSink Enhancements/Features for XMSSource


Introduction to Messaging Toolkit


What is Messaging Toolkit

A set of edge adapters which help to read/write to/from a queue or topic in WebSphere Message Queue

– XMSSource– XMSSink

Uses the JMS api's exposed by the IBM XMS client in C++ language


Brief Architecture Diagram

WebSphere MQ

Queue/Topic

Streams

XMS Source

XMS Sink

Read Message

Write Message

Create connection

Create connection

XMS

Create Producer/

Consumer

MQ Client

Uses


Use Cases


Real time campaign management

Let's take the case of a Large Bank which has lots of credit card customers. Imagine if someone performs a credit card purchase in a classy mall. We know where the transaction has been committed and the bank can offer various cross sell and up sell products. The transaction committed is captured by Streams and then Streams processes this and posts the message on to the Messaging queue layer. Unica further connects to the messaging queue and instantiates the campaign. This example illustrates how organisations are utilizing SOA technologies to integrate their enterprise systems and how messaging toolkit adapters fit into the eco- system seamlessly to respond in real time and increase efficiency and timeliness of the processes.


Tracking stolen vehicles

DMV issues licences to lots of vehicles. Electronic Toll Collection(ETC) allows for electronic collection of tolls by identifying cars and electronically debiting the account of registered car owners. Information from ETC can be used for smarter crime detection and prevention.– Tracking stolen cars: Assuming all toll information is continuously sent

and filtered against the stolen car list by Streams, when the stolen car passes a toll, it can be flagged and sent to BPM systems.

– Tracking stolen devices: When a device theft is reported to enforcers, an entry is made to systems containing details about stolen devices. When a car containing the stolen device passes a toll, it can be flagged and sent to BPM systems.

– Further we can use GeoSpatial (GIS) toolkit to find out the tentative location of the vehicle and based on the speed and the route which that vehicle took we can help in tracking it. Further this information can also be plotted in a map, and displayed to interested parties.


Operators


XMSSource

This operator reads data from a WebSphere MQ queue or a topic and pushes the data into Streams. The main functionality includes:– Zero input ports, One or Two output ports– Validates if the Streams application can connect to WebSphere MQ during

runtime– Validates that the queue/topic name specified in the initial context element

exists during runtime– Validates that all the Stream tuple elements which do not have an explicit values

assigned to them are present in the native schema and the datatype of these elements/attributes match during compile time

– Validates if the message class specified in the native schema is a valid MQ message class during compile time

– For Initial/ Transient connection failures, the operator tries to reconnect based on the reconnection policy specified by the user

– If an error occurs while reading the input from queue/topic, writes the error string into the error output port (if one is specified)

– On successful read, submits the tuple to the output stream corresponding to the message on MQ


XMSSink

Publishes data from Streams to a WebSphere MQ queue or a topic. The main functionality of this operator includes:– One input port, One output port(optional)– Validates if the Streams application can connect to WebSphere MQ during

runtime– Validates that the queue/topic name specified in the initial context element

exists during runtime– Validates that all the attributes in the native schema are present in the input

stream tuple and the datatype of these elements/attributes match during compile time

– Validates if the message class specified in the native schema is a valid MQ message class during compile time

– For initial/transient connection failures, the operator tries to reconnect based on the reconnection policy specified by the user

– If an error occurs while writing to a queue/topic, writes the error string and the erroneous input tuple into the error output port (if one is specified)

– On successful write submits the tuple in the input stream to the corresponding message on MQ


Terminologies and concepts


Connection Specification Document

xml file which specifies the details of connection to WebSphere MQ


Message class and native schema

Message class specifies how a Stream tuple is to be serialized into a MQ message. There are 7 types of message classes:– map : A message containing a collection of name/type/value triplets – stream : A message containing ordered list of type/value pairs – empty : An empty message– xml : A message containing a generic XML message– bytes :A Message containing a stream of binary data– wbe : A message contains an XML document in the WebSphere

Business Events event packet format– wbe22 : A message contains an XML document in the old WebSphere

Business Events event packet format 2.2 Native schema: Specifies the data attributes (their name,

type(external type) and length(optional)) for a message which is read from /written to a queue/topic in WebSphere MQ


Mapping between message class, native schema

Msg class XMSSink XMSSource

map Constructs XMS Mapmessage containing elements whose names and types are taken from native_schema and values taken from the operator input stream.

Expects a XMS MapMessage, containing elements whose names and types appear in the native_schema element.

stream Constructs XMS StreamMessage containing values taken from the input stream,serialized into the message in the order in which they are specified in the native_schema element, using the data types from that element.

Expects a XMS StreamMessage containing the values of the elements listed in the native_schema, in the order in which they are specified in the native_schema and with the data types specified in that schema.

bytes Constructs XMS BytesMessage containing values taken from the input stream, serialized into the message in the order in which they are specified in the native_schema element, using the data types from that element.

Expects a XMS BytesMessage, containing the values of the elements listed in the native_schema, in the order in which they are specified in the native_schema and with the data types specified in that schema.

empty Constructs an empty XMS message. Expects an empty XMS message.

xml Constructs a XMS message containing a generic XML message

Not Supported

wbe Constructs a XMS TextMessage containing an XML document in the wbe format.

Not Supported

wbe22 Constructs a XMS TextMessage containing an XML document in the wbe 2.2 format

Not Supported


Software Dependancies


Software Dependancies

Operating System: All Operating System supported by Streams except Power

IBM Message Service client for C/C++ (also known as XMS client) version 2.0.1 is required– The XMS client must be installed on the same machine as Streams is

installed and an environment variable (XMS_HOME) needs to be set to point to the install location of XMS Client

The WebSphere MQ Server and WebSphere MQ Client need to be at:– WebSphere MQ 7.0 + All Fix Pack 7.0.0.1 or later– WebSphere MQ 6.0 + Refresh Pack 6.0.1.1 or later– WebSphere MQ 7.1 is not supported by the current version of XMS :

2.0.1– The WebSphere MQ client libraries must be on the same machine as

Streams and an environment variable (MQ_HOME) needs to be set to point to the MQ Client libraries.


Features common to both Operators


Blob support for multiple message classes The SPADE version supported sending payloads in the message Blob support introduced for message class map, stream and bytes in both

operators Blob support introduced for message class xml in XMSSink as an

enhancement

<access_specifications>

<access_specification name="OutTrading1">

<destination identifier="Dest" delivery_mode="persistent" message_class="map" />

<uses_connection connection="Conn1"/>

<native_schema>

<attribute name="l" type="Bytes"/>

</native_schema>

</access_specification>

</access_specifications>



<destination identifier="Dest" delivery_mode="persistent" message_class="map" />


<native_schema>

<attribute name="l" type="Bytes"/>

</native_schema>



composite XMSSinkMain {

graph

stream<blob l> BlobData = FileSource(){

param

file : "Image.png";

format : block;

blockSize : 1276431234u;

}

() as Sink1 = XMSSink(BlobData){

param

connection : "Conn1";

access : "OutTrading1";

}

}


Support for Reconnection Policies

The default reconnectionPolicy in case of initial/transient connection failure was to try the connection infinitely

Support was introduced to allow the user to specify the reconnectionPolicy parameter with valid values of NoRetry, InfiniteRetry and BoundedRetry

Additional parameters reconnectionBound and period were introduced to give more flexibility to the user

Default is BoundedRetry with reconnectionBound of 5, period of 60 sec

composite XMSSourceMain {

graph

(stream<blob l>InputData) = XMSSource(){

param


access : "InTrading1";

reconnectionPolicy : BoundedRetry;

reconnectionBound : 10u;

period : 1.0;

}

}


graph


param



reconnectionPolicy : NoRetry;

}

}


graph


param



reconnectionPolicy : InfiniteRetry;

period : 1.0;

}

}


Support for an optional error output port Support was introduced in both the operators to write error string on

to the error output port in case of read/write failures Optional error output port is non-mutating and its punctuation mode

is free


graph

(stream<blob b>InputData; stream<rstring message> Error) = XMSSource(){

param



}

() as Sink1 = FileSink(InputData){

param

file : "XMSSourceOutput.txt";

flush : 1u;

}

() as Sink2 = FileSink(Error){

param

file : "XMSSourceError.txt";

flush : 1u;

}

}


Other Enhancements

Supporting external types in native schema– Previously SPL types were specified in the native schema– Changes to accommodate specifying external types in native schema

for ease of use

Introduction of environment variables to point to MQ and XMS install locations– The XMS install location was hard coded to /opt/IBM/XMS and the MQ

install location hard coded to /opt/mqm– Environment variables XMS_HOME and MQ_HOME introduced for

allowing flexibility

Support for uint8, uint16 and uint32 data types.– Support initially existent only for signed integers– Support for uintb data types introduced


Enhancements/Features for XMSSink


Negative length support on message class bytes

Message class bytes allows specification of -2,-4, and -8 as valid values in the native schema for string and blob

Negative length indicates that a 2,4 or 8 byte length needs to be pre-pended with the data representing the length of the data

composite XMSSinkMain {graphstream<rstring k>InputData = Beacon(){

paramiterations:1u;

outputInputData :k="ThisIsALongString";

}() as Sink1 = XMSSink(InputData){

paramconnection : "Conn1";access : "OutTrading1";

}

composite XMSSinkMain {graphstream<rstring k>InputData = Beacon(){

paramiterations:1u;

outputInputData :k="ThisIsALongString";

}() as Sink1 = XMSSink(InputData){

paramconnection : "Conn1";access : "OutTrading1";

}



<destination identifier="Dest" delivery_mode="persistent"

message_class="bytes" />


<native_schema>

<attribute name="k" type="String" length=”-4”/>

</native_schema>





<destination identifier="Dest" delivery_mode="persistent"

message_class="bytes" />


<native_schema>

<attribute name="k" type="String" length=”-4”/>

</native_schema>




Truncation and padding support for string When a positive length is specified in the native schema for

string in message classes map, stream and xml, it is treated as an upper bound for that string

For message class bytes, the string is truncated/padded in accordance with the length in the native schema

composite XMSSinkMain {graphstream<rstring k>InputData = Beacon(){ param

iterations:1u; output

InputData :k="ThisIsALongString";}() as Sink1 = XMSSink(InputData){ param

connection : "Conn1";access : "OutTrading1";

}

composite XMSSinkMain {graphstream<rstring k>InputData = Beacon(){ param

iterations:1u; output

InputData :k="ThisIsALongString";}() as Sink1 = XMSSink(InputData){ param

connection : "Conn1";access : "OutTrading1";

}



<destination identifier="Dest" delivery_mode="persistent" message_class="stream"/>


<native_schema>

<attribute name="k" type="String" length=”7”/>

</native_schema>





<destination identifier="Dest" delivery_mode="persistent" message_class="stream"/>


<native_schema>

<attribute name="k" type="String" length=”7”/>

</native_schema>




Performance Metrics Support

XMSSink has the following metrices– nTruncatedInserts: The number of tuples that had truncated attributes

while converting to a message– nFailedInserts: The number of failed inserts to the WebSphere MQ– nReconnectionAttempts: The number of reconnection attempts made

before a successful connection


Enhancements/Features for XMSSource


Support for initDelay parameter

Support for initDelay parameter introduced in XMSSource


graph


param



initDelay : 10.0;

}


param


flush : 1u;

}


graph


param



initDelay : 10.0;

}


param


flush : 1u;

}


Performance Metrics Support

No performance metrics was existent in the SPADE version

XMSource supports the following performance metrics– nMessagesRead: The number of messages read successfully– nMessagesDropped: The number of messages dropped – nReconnectionAttempts: The number of reconnection attempts made

before a successful connection



Documents

Messaging Toolkit InfoSphere Streams Version 3.0