Design patterns in application integration

Polish-Japanese Institute of Information Technology

Chair of Software Engineering

Master of Science Thesis

Design patternsin application integration

based on messages

by

Mariusz Pikula, Adam Siemion

Supervisor: Dr Piotr Habela

Warsaw, 2007

AbstractThis thesis is devoted to the issues connected with application integration. It presentsthe importance of the knowledge of the design patterns connected with this subject.At the beginning this thesis introduces the reader to the area of application integra-tion and different types of problems connected with it. The thesis explains why thetask of integration is being undertaken and why it can be very difficult and compli-cated. Next, the integration styles are being presented starting with the oldest andthe simplest ones going through more complex ones and ending on the integrationbased on messages, which forms the main area of interest in this thesis. After fa-miliarising the reader with the basics of this integration approach the thesis is aimedto provide him/her with the essential theoretical background, which covers knowl-edge about basic terms and design patterns connected with the integration basedon messages. Afterwards, the practical application of the discussed terms is beingshown based on the case study describing system integration issues. Last part of thisthesis is dedicated to the integration platform that has been created as an integralpart of the thesis. The description of this platform contains information about usedtechnologies, application architecture and an example of its usage based on the casestudy presented earlier.

Contents

Contents i

1 Introduction 71.1 Loose Coupling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.2 Case study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2 Integration Styles 172.1 Application integration . . . . . . . . . . . . . . . . . . . . . . . . 182.2 Application coupling . . . . . . . . . . . . . . . . . . . . . . . . . 182.3 Integration simplicity . . . . . . . . . . . . . . . . . . . . . . . . . 182.4 Integration technology . . . . . . . . . . . . . . . . . . . . . . . . 192.5 Data format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.6 Data timeliness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.7 Data or functionality . . . . . . . . . . . . . . . . . . . . . . . . . 202.8 Asynchronicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202.9 Styles of integration . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2.9.1 File Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . 222.9.2 Shared Database . . . . . . . . . . . . . . . . . . . . . . . 242.9.3 Remote Procedure Invocation . . . . . . . . . . . . . . . . 262.9.4 Messaging . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3 Messaging based systems 293.1 Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.2 Message Implementations . . . . . . . . . . . . . . . . . . . . . . 313.3 Message Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.4 Message Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.5 Message Transformation . . . . . . . . . . . . . . . . . . . . . . . 353.6 Message Endpoints . . . . . . . . . . . . . . . . . . . . . . . . . . 383.7 Synchronous and asynchronous communication . . . . . . . . . . . 39

4 Design patterns in the application integration 43

5 Enterprise Service Bus 515.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515.2 Message Oriented Middleware . . . . . . . . . . . . . . . . . . . . 52

i

ii CONTENTS

5.3 Tightly coupled interfaces . . . . . . . . . . . . . . . . . . . . . . 535.4 ESB aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.5 ESB capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555.6 ESB components . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.7 Open source ESB products . . . . . . . . . . . . . . . . . . . . . . 585.8 ESB integration patterns . . . . . . . . . . . . . . . . . . . . . . . 59

5.8.1 VETO pattern . . . . . . . . . . . . . . . . . . . . . . . . 595.8.2 VETOR pattern . . . . . . . . . . . . . . . . . . . . . . . 605.8.3 Two-step XRef pattern . . . . . . . . . . . . . . . . . . . . 605.8.4 Forward Cache Integration pattern . . . . . . . . . . . . . 62

6 Case study: Messaging systems work principles 65

7 Implementation 737.1 The origin of the name . . . . . . . . . . . . . . . . . . . . . . . . 737.2 Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737.3 Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 747.4 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 757.5 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 767.6 Processing sequence . . . . . . . . . . . . . . . . . . . . . . . . . . 777.7 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

7.7.1 Configuration model . . . . . . . . . . . . . . . . . . . . . 797.7.2 Configuration example . . . . . . . . . . . . . . . . . . . . 807.7.3 More on transformers and routers . . . . . . . . . . . . . . 817.7.4 Performing the configuration . . . . . . . . . . . . . . . . . 82

7.8 Integration design patterns supported by pESB . . . . . . . . . . 837.9 Problems encountered during the implementation . . . . . . . . . 84

8 Summary 85

Bibliography 87

List of Figures 91

List of Tables 93

Index 95

Acknowledgements

We would like to thank Dr Piotr Habela, who conducted a class on the softwareintegration (Technologie Internetu), which inspired us to perceive more informa-tion about this subject.

I, Adam Siemion, would also like to thank Remigiusz Weska, with whom I hadhave the pleasure to work and the company IMPAQ Sp. z o.o., where we — both— had been working on a project, which aimed to choose the best messagingbased integration solution that would fulfill the requirements of our customer.That experience also motivated me to delve into the subject of integration.

1

Preface

Business environment is a constantly changing volatile environment that requiresgreat flexibility from its participants. This flexibility requires the will to cooperatewith each other in order to maximise profits or obtain other types of benefits.The cooperation between two business entities involves information exchange andcollaboration of IT systems used by those entities. This collaboration might belimited only to the exchange of data or might be as complex as the usage of thepartners software functionality.

Combining two separate IT systems into one entity, capable of exchanginginformation and maintain a constant data flow between them, can be a verycomplicated task, especially in the business environment. This environment hasa lot of features that contribute heavily to the difficulty level of this task.

First of all, the system used by the business entities might not be an up-to-date IT system, but a legacy system designed in the early 90s or even earlier.Those systems can be very complex and vital for the business so that it is notpossible to simply replace them with new ones. Very often the complexity ofthose systems makes it too expensive to design, implement and introduce a newsystem with the same functionality as the previous one. The human factor alsohas to be taken into account, workers that have used the old system for a coupleof years might resist against the introduction of a new system that will replacethe one they are used to.

Moreover, business entities that want to cooperate might be located in distantgeographical locations. This fact adds additional issues into consideration. Issuessuch as communication reliability, security, communication errors handling, non-repudiation and so on. The reliability of the communication depends also onthe third party business entities such as external Internet providers, who areresponsible for maintaining the Internet connection. Thus, there are also externalfactors that have to be taken into account while coping with this problem.

Designing such a solution that would allow to combine the application usedby both business entities is another issue that must be considered. First of all,the designers of such a solution must have a deep knowledge about the businessprocesses in each of the entities in order to design an effective solution. Secondly,both of the interested entities must agree on such a solution. This can be difficultto achieve, because very often each of the business entity representative wouldput pressure on the designers so that the outcome solution would be based mainlyon their IT structure (internal data model, processes, etc.). Creating a design

3

4 PREFACE

that would satisfy all of the participants can be a demanding and difficult taskitself, not only because of this aspect, but the designers must also concentrate onthe functionality, flexibility and reliability of the solution, which makes this taskeven more difficult.

After the integration project is completed, the appropriate tool must be cho-sen to put the whole solution in motion. The IT market offers many availablesolutions that are specifically designed to solve those type of problems (i.e. build-ing integration solutions). We can choose from both open source solutions suchas the Mule or ServiceMix and propriety products from IBM, Oracle, Sonic Soft-ware, BEA and so on.

Upon taking a closer look at those products and considering the problems andchallenges of the application integration topic, we decided to make it our objectof interest and the topic of this thesis.

Having a limited amount of time and available resources we did not aimto create a solution that could compete with those made by large IT companies.Instead, we decided to take a different approach — create a lightweight integrationplatform. It would provide the basic functionality needed for an applicationintegration combined with the ease of use and a short learning curve, so thatthe potential user with a knowledge in programming would be able to effectivelycreate an integration solution without the need to sacrifice a large amount of timeto learn the functionality of the software program itself. The simplicity of ourplatform, in comparison to the large and complicated tools offered by large ITcompanies, would be its greatest strength. We aimed to create a tool, based onwidely available technologies, that could be used as a base for further developmentby adding extensions to it.

The work itself has been structured in such a way that a reader would getan overview of the whole topic of integration and especially messaging systemsbefore moving forward to the description of the created integration solution itself.

The thesis is divided into two main parts. Part one contains chapters two,three, four and five, which describe the basic theory behind the subject of integra-tion. While part two introduces the reader to the integration solutions currentlyavailable on the market such as Message Oriented Middleware (MOM) and En-terprise Service Bus (ESB), which are heavily using the concepts depicted in thepart one and presents a case study that is aimed to present the usage of termsand concepts presented in the first part of this thesis from a practical point ofview. Part two contains chapters six, seven and eight. The ninth and last chaptercontains the summary of our work.

1. Part one begins with an introduction to the integration topic in chapterone.

2. Chapter two concentrates on different integration styles, with the detaileddescription of each style along with its advantages and disadvantages aswell as the situations, when each of those styles might be used.

3. Chapter three concentrates on one particular integration style — the mes-saging. This style is the one that will be used in our own integrationsolution. In this chapter we will try to give a more detailed description of

PREFACE 5

this integration style. The description will cover two different concepts ofcommunication using this integration style — the synchronous and asyn-chronous one and basic concepts directly connected with this integrationstyle — Message, Message Channel, Message Router, etc.

4. Chapter four covers the concept of the design patterns in messaging systems.It gives the depiction of several selected design patterns with possible waysof usage and different variants that can be applied in different situations.

5. Part two begins with chapter five, which will introduce the concept of an En-terprise Service Bus (ESB). First, we will define briefly what an ESB is,then we will focus on the basis of the ESB — Message Oriented Middle-ware (MOM), advantages of introducing an ESB, its capabilities and finally,we will provide a couple of ESB integration patterns.

6. Chapter six presents a case study, which aims to present the usage of theconcepts from the previous chapters in a real life business example. It startswith a general overview of the problem and goes through all phases of theintegration process up to the final solution.

7. Chapter seven will be devoted to the description of the created integrationsolution. At the beginning, we will explain the concept of this approach tothe integration, the technology it is using and its architecture. To betterillustrate the way this system operates we will provide detailed sequencediagram with the detailed description of sequence of actions that are tak-ing place when two systems communicate using our solution. Also, wewould like the reader to be able to solve real integration problems using ourproduct, thus as an example of usage, we provide an imaginary integrationproblem with possible solution using our system.

8. Finally, chapter eight will contain the summary of our work with final re-marks regarding the goals that we managed to achieve and the ones thathave not been achieved, along with possible reasons why it has happened.It will also contain the suggestions about the possible ways of further de-velopment and some ideas of extensions that could be made to make theexisting tool more usable and enriched with new functionalities.

Chapter 1

Introduction

The task of integrating computer systems emerges as a response to a frequentneed to connect multiple separate computer systems. Integrated systems aresupposed to cooperate and provide unified functionality. Moreover, it is expectedthat the new — integrated — system will be operating on the data gathered by allof the participating computer systems. There are a lot of factors that make thisa difficult and challenging task. Systems, which are supposed to be integrated,might operate within different organisations, might be designed using differenttechnologies, might be legacy systems with no maintainers etc.

The need to integrate might arise because of multiple reasons, for example, themerger of two companies, the merger of multiple branches of one company, the aimto have one system responsible for coordination of others. Another reason mightbe cost reduction. One unified system is cheaper to support and more effective interms of business processing than two systems working separately [16]. Expectedresults of the integration are usually as follows:

• cost reduction

• increased effectiveness

• improvement of business processes

• unified flow of information

Applications, which are supposed to be integrated can be of different origin.They can either be custom made according to the customer specification to suitcustomer needs or bought as a commercial-off-the-self (COTS) application andtailored to meet the requirements of the buyer. The main differences betweenintegrated systems can also be of other kinds. They might be written in differentprogramming languages and be of a different age. One of those systems, or evenboth of them might be a, so called, legacy application [7] — computer softwarecreated couple of years ago, not being improved any longer, quite often with nodocumentation available and designed as a local standalone system. Usually,those systems provide no communication capabilities. Network communication

7

8 CHAPTER 1. INTRODUCTION

and data exchange have not been taken into consideration during the design ofthose kind of systems.

What makes the process of integration even more difficult is the fact that:

• systems which are to be integrated can be spread geographically with ma-chines, on which they are running, placed in distant locations

• systems might be written in different programming languages

• systems might not have any documentation

• systems might be running in different environments (operating systems,hardware configurations, etc.)

• systems might be managed by different organisational units or even com-panies

• companies, which provide proprietary software, are not willing to partic-ipate in the integration process, because they do not want to reveal theinternals of their products

• workers may not be willing to adapt to changes made by the integrationprocess

Reliable communication between integrated systems must be assured usingavailable communication facilities. The integration should be introduced withoutmaking too many changes to any of the existing systems. There is a very im-portant reason behind this limitation. Computer systems such as those runningwithin some financial institutions and managing data crucial for its activities can-not be redesigned and reimplemented from the scratch. It would be too expensiveand risky for the business. The complexity and scale of those systems makes theprocesses of recreating them costing too much for an institution to afford it.

All those problems and difficulties must be taken into consideration whenan integration solution is being designed for a computer system. Because ofthe above reasons, the integration of multiple various computer systems can beconsidered as an especially difficult and challenging task.

The aim of this thesis is to present the subject of an application integra-tion from the theoretical point of view and use the presented knowledge to builda lightweight easy-configurable solution that will allow to integrate different sys-tems in a quick and efficient way. The goals that are to be achieved and the basicconcept of this system have been outlined in the first chapter of this thesis. Be-fore going further on to the system architecture description, the way it has beendesigned and implemented along with the detailed description of its concepts andgoals that have been set during the process of creating it, a brief overview andbackground on the subject of system integration will be given. This overviewshould allow the reader to get an overall view on the topic and gain knowledgeabout the concepts and terms used through this document such as loose cou-pling, design patterns and others that are essential for understanding the basicsof a system integration.

CHAPTER 1. INTRODUCTION 9

1.1 Loose Coupling

Loose coupling [11] has recently become a very popular term directly connectedwith application integration. It can be said that loose coupling is one of the keyconcepts formed around this topic. This strong relationship makes it necessaryto explain this term prior to moving forward to the next chapters. This will allowto have a full understanding of the topic, which is being described in this thesis.

The main concept of loose coupling is that two communicating parties (sys-tems or applications) should make minimal assumptions about each other. Inother words, the less applications need to know about each other in order tocooperate properly, the better. Applying this principle should, in the long term,reduce the costs of maintaining an integration solution, make it more effectiveand reduce the costs of changes of integrated applications.

Loosely coupled applications can be modified independently (to some degreeof course), which means that changes made within one application do not enforcechanges in the coupled application. Therefore, using loose coupling in applica-tion integration makes an integration solution more flexible and change tolerant.This flexibility derives from the fact that connected applications do not have tobe adjusted after changes done in on of the systems taking part in the communi-cation.

If processing of one of the applications is based on the information aboutthe internal business logic of the other application (e.g. data format), then anychanges made to that logic will automatically enforce the changes within otherapplication. Thus, the less dependent, on this type of information about eachother, the applications are, the more flexible communication between them canbe maintained.

Application coupling is a multidimensional issue that covers not only theissues connected with the design and implementation of the application. Havingthat in mind it seems obvious that loose coupling between applications mustbe viewed from different aspects. For example, integrated applications can beloosely coupled in time — they do not have to be working at the same time tocooperate — using queues. They might also be loosely coupled in format — everysystem might have a different data model — using a component responsible fortransforming messages exchanged between the applications.

Measuring the degree of loose coupling is a separate problem. When an in-tegration solution is being designed with the concept of loose coupling in mind,it would be desired to have the means to measure how loosely coupled the inte-grated applications really are. In order to be able to do this a measurable featureof both applications must be found. Such a feature should allow to conducta measurement and give results that can be compared. The feature that couldbe used for this purpose is the number of changes that can be made within anintegrated system without the need to interfere with the integration solution asa whole. Changes can concern business processing within an application, dataformat and so on. The more changes can be made within an application withoutthe need of altering the solution itself, the higher the degree of loose coupling is.Of course, many measures can be thought of and used. But this one (number ofchanges) concentrates on the key concept of loose coupling and from that point


of view, it might be considered as the best one to use. Moreover, this measureemphasises the main idea of loose coupling that the integrated systems shouldmake as little assumptions as possible about each other. The fewer assumptionswill be made, the more changes can be made inside the connected system withoutaffecting the operation of others.

The opposite concept of loose coupling is tight coupling. Tight couplingmight be depicted using local method invocation as an example. Local methodcall imposes a lot of assumptions on the caller, which are as the following:

• called method must be written in the same programming language as thecalling method

• the exact number and type of the arguments of the called method must beknown

• called method must run in the same process

• both calling and called method must use the same data types format

• both calling and called method must use the same internal data represen-tation format

The consequence of those assumptions is that tightly coupled local methodinvocation differs in many ways from loosely coupled communication, based onthe messages for example. First of all, local invocation starts to perform its ac-tivities immediately after receiving a call from the calling method, thus there isno latency between the method call and start of the processing. Local methodinvocation is very fast, efficient and reliable in comparison to a remote commu-nication. Although, the calling method must wait for the called method to finishits processing and return the result (synchronous method call), the processingspeed is much greater than in the case of a remote communication. In case ofa remote call waiting time increases because of the distance, latency of the con-nection between the applications, connection medium quality and so on. Becauseof the communication speed and the fact that the method invocation is local, thesecurity and communication issues (security, reliability, performance, efficiency)that would come up in case of a remote call, are not a problems that have to beworried about. When a local call is being invoked, it is certain that it will reachits destination and will be performed. In case of a remote call there is no guar-antee that the call has reached its destination. There might be a communicationfailure, connection might be broken, the remote host might be out of order oreven the data can be altered while being transferred from the sender to the caller.All those problems make a remote call far less reliable than a local call.

As it can be easily noticed, local — tightly coupled — method invocation issimpler, more efficient and generates far less issues to worry about. The sourceof this efficiency and simplicity lays in the assumptions made by the applicationsabout each other.

In order to make integration easier, many communication technologies use thesame semantics as a tightly coupled local method calls, to invoke their function-ality and exchange data between applications. Those technologies include:


• .NET Remoting [15]

• Java Remote Method Invocation API (Java RMI) [8]

The main advantage of this approach is that it is easier for the developers, whoare used to invoking local methods, to start using these technologies. Therefore,using those techniques may lead to making the same assumptions as in the caseof local method invocation. However, it should be kept in mind that while thoseassumptions are valid in local environment, many of them are not valid in the caseof remote calls (making them valid, if possible, will greatly restrict the flexibilityof the integration solution).

When integrating applications it is not usually desired for a calling applica-tion to wait until the results of the remote processing will be available. Sucha waiting, at best, might lead to delays that are very often not acceptable bythe business entities taking part in the integration process. Moreover, in caseof a communication failure (e.g. due to lost of connectivity) an application canget suspended waiting for the response. This can lead to an application crash.The processing time of a remote call is also much longer than in the case of a localcall, and the call as a whole is far less reliable. When a remote call is being madeit cannot be assumed, as in the case of local call, that a response to this call willbe received, because there might be communication failure, crash of the remotesystem, and so on.

One of the assumptions made in the case of tight coupling is that both calledand the calling method are written in the same programming language. This as-sumption significantly reduces the scope of possible integration scenarios, becauseit is only possible to integrate applications written in the same programming lan-guage (e.g. it is not possible to integrate applications written in Java and C#using JAVA RMI). This restriction highly reduces the flexibility and scope ofapplications of the technologies mentioned above. What is more, it makes it im-possible to integrate newly written systems with the legacy systems. As it canbe easily seen, this approach is also burdened with problems that do not occurin the case of tightly coupled local calls.An example showing what problems canappear while trying to integrate systems with tightly coupled dependencies canbe found in [6], along with the detailed description of this approach and problemsthat it might cause.

Loose coupling apart from being a popular term is also one of the core conceptsof an application integration. By making integrated system less dependent fromthe things such as the programming language in which they are written, their datamodel, internal business logic and architecture they are more flexible and changetolerant. This approach allows to modify one application to some point withoutthe negative effects on the communication with the other system. This assuresflexibility, which cannot be achieved in case of tight coupling due to restrictionsmentioned earlier.

Apart from such benefits as flexibility, scalability, higher tolerance for internalchanges loose coupling has also some disadvantages. Designing a loosely coupledintegration solution is a more complex task than in the case of tightly coupledsolutions. A lot of new problems need to be solved in order to effectively perform


a loosely coupled integration solution. The difficulty in designing such a solutionalso results in more difficult development, error tracing, debugging, etc.

To sum it up, loose coupling minimises the interdependency among systemsin terms of time, information format, and technology at the cost of more sophis-ticated design and implementation.

Now, since the loose coupling term and its role in the application integrationhas been covered, let us move on to a more detailed description of differentintegration styles along with the benefits they bring and — of course — threats,which might occur while using them in integration solutions. But before thata case study will be presented as a practical illustration of the discussed issues.

1.2 Case study

An example will be discussed now in order to illustrate all those informationprovided in the above paragraph. The example will cover a simple case study,which covers banking application integrated with the front-end Web application.This application will allow users to transfer money between user accounts. Theapplications will communicate using the TCP/IP protocol stack, which is themost common and wide spread set of communication protocols.

The presented source code snippet accepts the following information on itsinput:

• transfer title

• destination account number

• transfer amount

The sample source code snippet, written in C#, could look like this:

String hostName = "www.mybankingapp.com";int port = 8080;

IPHostEntry hostInfo = Dns.GetHostByName(hostName);IPAddress address = hostInfo.AddressList[0];IPEndPoint endpoint = new IPEndPoint(address, port);

Socket socket = new Socket(address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);socket.Connect(endpoint);

byte[] amount = BitConverter.GetBytes(1000);byte[] transferTitle = Encoding.ASCII.GetBytes("My Transfer");byte[] destAccNumber = Encoding.ASCII.GetBytes("1234445321234321");

int bytesSent = socket.Send(amount);bytesSent += socket.Send(transferTitle);bytesSent+=socket.Send(destAccNumber);

socket.Close();

Above source code excerpt first initiates the connection to the banking system,then sets an amount of money that will be transferred to the destination accountalong with the transfer title and the destination account number and finally sendsthat information as a byte stream. Of course in the real life this method would


be much more sophisticated, but the goal here is to show general concept, not towrite a complete business solution.

The communication solution presented above is quite straightforward andsimple. It does not require the usage of any sophisticated integration software.But this solution carries hidden problems that can be very hard to track andrepair.

In multiple books about network programming the above solution would bepresented as the one, which enables to communicate the client (presented above)with the server regardless of the operating system and programming languagethese two systems are using. This is not completely true, as it will be explainedlater.

In order to obtain data that will be sent to the banking application, thetransfer amount, transfer title and destination account number are converted toarrays of bytes. Then each of them is sent to the destination. The BitConverterclass is used to convert transfer amount to the array of bytes. The conversionmade by this class is performed using internal memory representation of a givendata type (integer in this case). .NET uses 32-bits integer type and this type willbe used in this case to make the conversion of integer to a array of bytes. Othersystems might use not a 32 bit representation, but a 64 one for example. In caseof a system using 64 bit representation it will read not 32 but 64 bits from theincoming byte stream. What does this mean in the case of our example? If thedestination system uses the 64 bit integer representation it will read not only the4 bytes of the transfer amount but also the preceding 4 bytes of the transfer titleand try to interpret whole 8 bytes as an integer. This difference in data typeswould cause a different amount of money being transferred than the user hadinitially desired! Apart from that, for the same reason the destination accountnumber would be different than the one given by the user. Such a behaviour isat least undesirable and will lead to the disastrous effects both for the bank andthe client.

Moreover, client and bank computer systems may use different formats tostore numbers. One of them may use big-endian system, which stores numbersstarting with the highest byte first, while the other one may use the small-endiansystem, which stores numbers starting with the lowest bytes first. This will alsocause difference in the transfer amount!

At least two assumptions must be made about integrated systems in orderfor the above solution to work properly. First one that both of them have to usethe same data types, and the second one that both of them have to use the sameinternal number storage format. However, it is not the end of the restrictionsimposed on by this approach.

Upon the closer examination of the above source code, a couple of things mightbe spotted. First of all, the connection information have been written directly inthe source code. Any change concerning this data, like changing the destinationhost name or adding an alternate destination address would require altering thesource code. In order to take effect of those changes the whole application wouldhave to be recompiled and redeployed. In the case of a simple application itmight not appear as a difficulty, but when more complex, critical applicationsare being concerned such a way of performing changes can become a very serious


issue. It would significantly increase the cost and time needed to introduce eventhe simplest change to the application. The above source code should be writtenin such a way that changes could be made to it in the most efficient possible way(efficiency in that case covers both time and cost efficiency alike).

Furthermore, the usage of client-and-server mode assumes that both the serverand the client are connected to the network at the same time. If one of theparticipants is currently not available, because of network problems, too highnetwork traffic, connection link problems, etc., then the connection cannot beestablished and the data cannot be exchanged.

It has already been mentioned before that in case of the presented solutionany changes made to the application require changes within the application codeitself. Those changes would have to be made each time a destination of the requestbeing sent would change (that would involve changing the code, recompiling itand redeploying application).

The same way of introducing changes to the application — changing thesource code, recompiling it and redeploying — would also have to be taken alsoif there would be a need to change the number of parameters being sent to thebanking application. But this time the changes would have to take place in bothapplications, because the banking application needs to know in advance the exactstructure of the request so it can parse and process it correctly.

This example shows how the tightly coupled solution could look like and whatassumptions must be made in order for it to work properly. To sum it up, thoseassumptions are as follows:

1. Both client and host system must use the same internal number formatrepresentation and use the same data types.

2. Both client and host applications must be working and connected to thenetwork at the same time in order to exchange data.

3. A client application must know the host location during the coding phaseand this location cannot be changed without updating the application codeitself.

4. A host application must know the exact number and type of the requestparameters during the coding phase in order to process and interpret themcorrectly. Changes can only be made by changing the host application’scode itself.

As it has been shown, a lot of assumptions must be made about the appli-cations in order to make them communicate correctly. Therefore, the presentedsolution can be qualified as a tightly coupled and makes a good illustration ofrestrictions and limitations of this approach. In order to make this solution moreflexible and less restricted it should be designed as a loosely coupled. Redesign-ing it to achieve that goal, would mean removing the restrictions limiting theflexibility of the presented solution. That goal would be achieved if there wouldno longer be a need for all those conditions, listed earlier, to take place in orderto make the systems fully functional.


The first step on the way of decoupling the previous solution would be defininga platform independent data format, which would be resistant to issues connectedwith different internal format number representations and the usage of differentdata types. An XML can be a solution of this problem, it can be used to definerequest description format, then requests would be sent as XML documents.The destination host could parse it and extract all the necessary information toprocess the request.

In order to remove the restriction caused by the the assumption concerning thehost location and communication issues, a Message Channel design pattern (3.3)can be introduced. Message Channel is a logical address, which both the clientand the host application use to communicate. The application has to be able toconnect to the channel only, not directly to the server. This resolves the locationissue — there is no longer the need to know the connection details about the hostapplication — the channel is used to communicate.

If the channel will be able to store the requests in form of a request queue,then the necessity for both systems to be connected at the same time will beeliminated. Every request will be stored in the channel until the destinationsystem fetches it. The response delivery will work the same way. Thanks to that,the systems will be able to communicate without the requirement for both ofthem to be on-line at the same time.

Applying the above mechanisms to the given problem would change the so-lution from a tightly coupled into a loosely coupled. With those mechanisms asolution — far more flexible than the previous one — can be obtained, free of allthe restrictions of the first approach. From now on the client and the host appli-cation might be developed simultaneously, independently of each other. Changesdone in one participant would not require altering the other one.

However, there are also disadvantages of loosely coupled solutions — the maindrawback is that it becomes much more sophisticated and complicated. Thismeans that it would take much more lines of code to implement this solution,and it would not be so simple and straightforward. Also the process of debuggingand testing becomes more complex. It is very important to keep that in mindwhile making the decision between loose and tight coupling model.

The above example forms a good illustration of the differences between thetight and loose coupling. It shows what both of them have to offer and whattheir advantages and drawbacks are.

Chapter 2

Integration Styles

As mentioned before, application integration is a task of making two or moreseparate systems to work as one - combined - system and share functionality.Problems usually faced while integrating applications have been briefly describedin the Introduction chapter . In this chapter we will describe different integrationstyles that might be chosen when coping with this challenging task.

Every integration problem is different, therefore the choice of an integrationstyle must depend on the circumstances of a particular situation. In order tochoose an appropriate style - criteria, on which the choice will be made, must bedefined. Martin Flower [7], for that purpose, provides the following:

• Application integration

• Application coupling

• Integration simplicity

• Integration technology

• Data format

• Data timeliness

• Data functionality

• Asynchronicity

When the integration task at hand will be examined and analysed based onthose criteria. Conclusion that will be the result of such an analysis can be veryhelpful later on. Basing on those conclusions, the decision on, which integrationstyle will be the most suitable for the given situation, can be made. Beforemoving forward to the description of those styles, a brief overview of each ofthe mentioned criteria will be given. This will allow the reader to have a betterunderstanding of those criteria and enable applying them in a proper way.

17

18 CHAPTER 2. INTEGRATION STYLES

2.1 Application integration

Before starting to think about applying an integration solution one must thinkabout whether it is possible to develop a new, centralised local application insteadof integrating separate already-existing systems. A local application will be easierto design and develop with far less issues, such as security, communication, dataexchange, etc., to consider. Of course this might only be done when it canbe afforded to implement a new application from the scratch. As it has beensaid before, in many cases it is not only too expensive but also impossible torecreate the system again from the beginning. Thus, there is no other way thanto integrate existing systems. However, the necessity of integration should bealways thought trough. It should be undertaken only if it is necessary. If there isanother way of solving the problem, in a less expensive and more effective way,it should be considered before.

2.2 Application coupling

The application coupling has been discussed in details earlier (1.1). The mainrule, which should be kept in mind is that while choosing an integration solutionone should try to pick the one that will enable to achieve the greatest decouplingof the integrated systems (sample way of measuring the decoupling level hasbeen described earlier). This will ensure that the systems will be more flexible.Flexibility allows to make changes within the application without the need tochange the whole integration solution and the way applications communicateand exchange data between each other. In a long time perspective this featuremight appear much desired. It will allow to upgrade one of the systems or enrichit with new functionality without the need to change the remaining integratedapplications.

2.3 Integration simplicity

When deciding, which integration style is the most appropriate, the simplicity ofthe integration solution should be one of the most important factors being takeninto consideration. It is defined not only by the complexity of the design andused technologies, but also by the number of changes that need to be done inintegrated applications in order to integrate them into one system with sharedfunctionality. Although, on the other hand, the simplicity should not be enforcedat the cost of integration quality or the functionality of integrated systems.

The main effort should be put into making the integration process as simple aspossible and limiting the changes that need to be done in integrated applications.While, at the same time, the desired functionality that has to be achieved andthe overall quality of the integration solution should be maintained. However, itis worth keeping in mind that sometimes it is better to choose a more complexsolution in exchange of a higher quality and flexibility of the final result.

CHAPTER 2. INTEGRATION STYLES 19

2.4 Integration technology

The market of integration technologies is constantly evolving, new products andstandards are being introduced all the time, thus choosing the most up-to-datetechnology can be tempting, but it is not always the best thing to do. The costsof introducing a new technology must be considered in the first place. Productsfeaturing state-of-the-art technologies are not always the ones that suit needs inthe best way. What is more, deploying a product using elaborate technology intoa company requires additional time to master it and gain some experience. Timespent on training consumes additional resources and increases overall cost of theproject. This extra time might not always be available. Also, the failure risk riseswhen the development team have to use the technology they are not experiencedwith. Furthermore, new solution that has just emerged can be unstable, nottested sufficiently, inefficient, unreliable, etc. Therefore, introducing the newesttechnologies into the solution can not only significantly raise the overall cost ofthe integration, but also - what is more important - raise the risk of integrationfailure. All those things mentioned above should be taken into considerationwhen choosing what technologies to use to perform integration.

2.5 Data format

Integrated applications need to exchange data in order to perform their activities,data that will be legible to all of the participating applications. To fulfil thisrequirement unified a data format, readable for all of the integrated applications,must be set. This can be achieved by formatting data to one common formatwithin the integrated applications or by using a component (translator), whichwill translate data to an appropriate format and ensure it is readable by thedata receiver. The first option may be considered as a worse one, because itrequires more code modifications within the integrated applications, so it maycollide with the Integration Simplicity criteria. When choosing the common dataexchange format, it should also be taken into account that the nature and formatof data produced by the integrated applications might change over time andthat those changes will affect the integration solution. This means that thetransformation of this data to the common data format will also change. Becauseof this, an integration solution should be flexible and able to adapt to data formatchanges within integrated applications.

2.6 Data timeliness

Another factor that has to be considered when choosing an integration techniqueis time. In particular, the time period from the moment data is being publishedto the moment when published data is being consumed by another application.This is one of the main factors that has the influence on the overall system per-formance. An effort should be put into minimising the time needed to exchangedata between applications. The shorter this time period will be, the faster inte-grated application can receive data, start processing it and return results back to


the data sender. In the case of the synchronous communication scenario, shortdata exchange time is essential to prevent delays in the application processing.

Moreover, long delays may cause another problem. The data, while beingtransferred to the destination, may become stale. In this situation the processingof this data can lead to errors that may have serious consequences for the business.This issue, named the data timeliness issue, is especially important in case ofapplications that deal with volatile data, which changes very frequently in ashort periods of time. Delays are not the only threat connected with this issue.A so called deadlock situation, may also occur when the sender is waiting toreceive the result of the processing from the receiver and the receiver applicationis out of order, because it had just crashed or because of other reason, it cannotcurrently process the sender’s request. In that case the whole system is suspendedand cannot perform its activities.

The mentioned issues make the integration process even more complex andshould be taken into consideration, when choosing the most appropriate inte-gration technique. The solution, which in given circumstances will provide theshortest latency, should be chosen. This should prevent communication and pro-cessing deadlocks and errors caused by the stale data.

2.7 Data or functionality

When deciding which integration technique to use, another thing should also beconsidered. It must be decided if the applications will share data or functional-ity. Sharing of functionalists makes an integration more complicated than justsharing of data. It is more difficult to design and implement and has a signif-icant impact on the integration process. It requires different approaches andtechniques than data sharing and is harder to achieve. The difference betweenthose two approaches has been described later on in this chapter, in the section”Shared Database” (2.9.2) and ”Remote Procedure Invocation” (2.9.3) of ”Stylesof integration”.

2.8 Asynchronicity

Another issue, which should be thought through while designing integration so-lution is the way, in which the integrated applications will communicate. Thereare two possibilities:

• synchronous communication

• asynchronous communication

In a synchronous communication scenario an application invokes the func-tionality of the remote system and waits until the remote system will process therequest and return results. After receiving the results it continues to perform itsactivities [2].

In case of an asynchronous communication scenario an application continuesto perform its activities right after sending a request and does not wait for the


results from the remote system. When the results finally come the sender systemis notified then it postpones its current activities and processes the response fromthe remote system. This type of call - the sender does not need the results ofthe request and it is the asynchronous call - is named ”Fire and Forget” and isa design pattern [5] used in the application integration.

Synchronous communication is simpler to design and implement, but reducesthe overall performance of the system. Also, it may cause deadlocks (see theabove section ”Data timeliness” (2.6)) and increase time spent by the system inthe idle state - waiting for the response from the other system. Asynchronouscommunication, on the other hand, is more effective and offers a greater perfor-mance at the cost of greater design and implementation complexity.

Synchronous communication is more suitable in cases when there is no needto create a complex solution, when messages being sent are small and the process-ing time is negligible. In that case the delays caused by waiting for the responsewill not affect the overall performance of the application in a noticeable degree.In other cases when an application needs to maintain the request-response inter-action model in order to provide desired functionality (e.g. web browsers, onlinechats, etc.), a synchronous model of communication is also necessary.

Asynchronous communication can be used when the sender does not expectthe response to arrive right away. Amazingly, this situation occurs very frequentlyin the real life. For example, after filling in the form for a VISA, we do not expectthe embassy to examine our application before we leave the building. Also, weare not waiting inside the embassy for the decision. Instead, we can continue withour lives. Similarly, after sending a letter in the post office, we do not expect thepost office to deliver the letter before we leave. Those analogies are very similarto the working of the asynchronous model. They are proving that this method ofcommunication is very popular in the real life, therefore it also has to be availablein the computer software.

Asynchronous communication might be more efficient, especially when thereis no place for delays, caused by the waiting for a reply of the request. An asyn-chronous application instead of waiting for the response, as it is in case of a syn-chronous one, might continue its processing. Although, it requires solving someadditional issues, such as the ability to process the data received in a responsefor the request sent earlier, what complicate the design of an asynchronous ap-plication, it may significantly improve its performance.

Moreover, the asynchronous communication is more reliable than the syn-chronous one. Because the asynchronous model usually involves the usage ofqueues, which can store persistently every received message, it guarantees thatno message will ever be lost. Even if the receiver system is currently not operat-ing, the queue will store all message designated for it, and when the system willbe online again it will fetch all of them.

Another advantage of the asynchronous model is the fact that it enables tocreate systems more resistant to high-loads. The difference between those twomodels is the way they behave during high traffic. In such a situation a syn-chronous application would not be able to provide a service to all clients, someof them would get an error, some of them would get no response at all, finallyeven the whole application could become inaccessible. An asynchronous applica-


feature synchronous asynchronousefficiency - +reliability - +resistance to communication errors - +resistance to high-loads - +design & implement difficulty + -

Table 2.1: Trade-offs between synchronous and asynchronous model

tion, on the other hand, would statistically process each request longer, becauseit would have a lot of requests waiting in the queue, but sooner or later each ofthem would be processed, no request would be left without a response.

The trade-offs between synchronous and asynchronous model has been summedup in the table 2.1.

2.9 Styles of integration

There are a few different styles of integration available to the developers facedwith the problem of integrating computer systems. They vary depending on thecomplication level, difficulty of design and implementation and so on. Criteriapresented in the previous sections can be helpful in determining which of thosestyles to choose to solve an integration problem. Each of those styles addressessome criteria better then the rest of them. Integration approaches can be groupedinto four main types. Those integration styles are as follows:

• File Transfer

• Shared Database

• Remote Procedure Invocation

• Messaging

Each of those techniques has been developed to handle the same task - theapplication integration. Although, the task remains the same, the approach rep-resented by each of them is different. Every one of them is more sophisticatedthan its predecessor (e.g. the Shared Database is more complicated solution thanthe File Transfer). When faced with the application integration task the point isnot to use the same technique in all cases, but to be flexible and basing on thecriteria described above choose the most suitable style for a given task. Morethan one style can be used to achieve the best final result. As mentioned before,Messaging would be the style on which this thesis will concentrate, but otherstyles will also be briefly described to give a wider scope of possibilities at hand.

2.9.1 File Transfer

The File Transfer (Figure 2.1) is the simplest integration style. The mainidea behind this technique is to use files as a data transfer mechanism between


Figure 2.1: File Transfer style(source: Enterprise Integration Patterns [7])

applications. Because files exists on every operating system and almost everyprogramming language has files operations, it makes them very universal solutionfor the purpose of information exchange. Also, as using files does not require anyadditional integration tools and as they are already available, they might seemto be an obvious solution, but they do have lot of disadvantages.

What is required in order to integrate application using this technique isan agreement on the format of the file used to exchange data. In most cases twospecial components (Figure 2.1), usually designed by the integration team, arecreated for that purpose:

• Export - component responsible for putting the data from the Applica-tion A into the file (according to the file format)

• Import - component responsible for reading and parsing the data from thefile (according to the file format) and inserting that data into the Applica-tion B

Apart from the file format another arrangements must be made. Namingconvention for the files has to be agreed, so that the file names remain unique (itshould be impossible for two separate files to have the same names). This is veryimportant in order to avoid name conflicts and situations, when the file with olddata would be processed.

Another important issue is to decide when the files will be written and read.Creating and processing such a file too often will burden an application unnec-essarily. Usually some fixed time periods are set based on the business activitycycles, e.g. files can be created on daily or weakly basis. Basing on those timeperiods the recipient application (Application B) checks if there is a new fileavailable to process. If too large time period is set then the application could bedesynchronised and errors in data processing could arise, because by the time thedata in a shared file would be consumed by the second application, they couldbecome stale and processing them might lead to errors.

When a file is created and data is being written to it, the lock mechanismmust be set in order to make sure that the other application is not trying to


Figure 2.2: Shared Database style(source: Enterprise Integration Patterns [7])

access the same file. This issue is also very important and should be taken careof in order to prevent errors while reading data from a file (e.g. unexpected endof a file).

Application using the File Transfer technique can be modified without af-fecting each other, because it has components, responsible for the export andthe import, separated from the application itself. Also because they only needto access the file containing exchanged data, no knowledge about the internalprocessing performed in each of them is required (such as method return types,method names, number and types of parameters passed to method and so on).

One of the main disadvantages of that technique is the fact that data is beingsynchronised in a batch mode. There might be situations when data processed byApplication B is no longer valid, but because synchronisation is taking place notin realtime but in time periods, Application B is not aware of that fact, until thenext synchronisation process. This excludes this type of integration in certainsituations, e.g. checking current bank account balance.

2.9.2 Shared Database

The Shared Database (Figure 2.2) is another integration style. It is a morecomplex and sophisticated approach than the File Transfer. The idea of thisapproach is to use a central data store that all applications share and can accessat any time.

This approach overcomes the main drawback of the File Transfer style, i.e. thelack of timeliness (data is not being available at proper time due to the fact thatis being exchanged in a batch mode). As explained before, files with data arecreated repeatedly at some fixed amount of time, so this solution is not applicable


in situations where data has to be propagated to all applications in realtime.Shared Database style does not have this disadvantage.

Moreover, in case when each application changes shared data very frequently(couple of times per second) the usage of File Transfer would be very inefficientand would lead to many problems. If integrated applications use the shareddatabase then even frequent data changes are not an issue, because data changedby one system is instantly available to the others.

Database engine is used to handle transaction issues in order to prevent anydeadlocks and data inconsistencies. When an error in data appears it is easier torollback a transaction than to return the application to the state prior processingof a file in the File Transfer integration style. Moreover, small but frequentchanges make it much easier to fix possible errors, without loosing too much ofthe processed information, than in the case of one huge daily or weekly basedupdate.

Almost every programming language gives the means to work with the rela-tional database using SQL queries. Also, there are a lot of tools available on themarket that make working with the databases simple and effective. The abovearguments might be considered as advantages of using a Shared Database, butthere are are also disadvantages of this solution.

When using a single shared database all of the integrated applications haveto produce data compatible with database schema. In order to achieve that theymust be modified. Designing the database schema, in this case, is the mostdifficult part of the whole integration. All participating departments, companies,etc. have to agree on one common schema. This can be a very challenging task,especially when each of the participants wants to save some parts of their ownschema, the one they got used to.

After the schema is set the existing data must be transformed, without a lossof any information and loaded into the new database. Data transformation canalso be a very complex process, because in some cases the existing data has to beheavily altered before it will become compatible with the new schema. Moreover,each of the participants has to have a very good knowledge about the model ofthe data used by his/her system. As experience shows, this is not always the case,because some systems might have been created many years ago, there might besystems without documentation and so on.

To sum it up, changes within applications can be made and they will noteffect the integration solution as long as the output data of the application iscompatible with the database schema.

Apart from advantages that come from overcoming the drawbacks of the pre-vious approach, this integration style has also some disadvantages, which areworth mentioning. The usage of one shared database by multiple systems cancause some serious performance issues or even lead to deadlocks. When twoor more applications are trying to modify the same data simultaneously, a dead-lock might occur because one application will place a lock on the accessed datathat will prevent other applications from modifying it at the same time.

Moreover, having one database used by all the applications incorporates single-point-of-failure. If the database is not operating then none of the applicationswill be able to perform their work.


Figure 2.3: Remote Procedure Invocation style(source: Enterprise Integration Patterns [7])

2.9.3 Remote Procedure Invocation

The Remote Procedure Invocation (Figure 2.3) represents yet a more so-phisticated approach to the integration problem than the two previous styles.Similarly, as in the case of shared database, the increased level of sophisticationenables to overcome the drawbacks of the two previous approaches.

The Remote Procedure Invocation is a mechanism that allows one applicationto invoke the method in the context of another - remote - application. Alongwith the invocation of an appropriate method all of the required informationis being passed on. The remote party returns the result after processing theinvocation [8]. The result might be as simple as a Boolean value, indicatingwhether the operation was successful or not or as complex as data structurecontaining the information about the customer.

In this approach, when one application needs data owned by another applica-tion it makes a direct call to that application. If data needs to be modified it isalso done by a direct call to the application that owns this data. Each applicationmanages its own data. No data is being duplicated in multiple systems, as it is incase of a File Transfer. Also, there is no need to change integrated applications incase the data, managed by one of them, changes. If there is a need to introducea new way of data processing than a new remote method has to be implemented.Although this approach requires the integration team to agree, in advance, on thenames of the methods that would be available for remote invocations, what kindof data would be passed with an invocation, what information would be returnedand so on.

Prior invoking remote method some knowledge about the other party is re-quired:

• number and types of arguments

• type of the result


Figure 2.4: Messaging style(source: Enterprise Integration Patterns [7])

• how error situations are handled: is the exception being thrown or somenegative value is being returned?

There are many existing implementations of Remote Procedure Invocation,just to name a few - CORBA, DCOM, .NET Remoting and JAVA RMI. As men-tioned in the previous part of this paper covering loose coupling, those implemen-tations use the same syntax and semantics as the local method calls. This makeit easier for the developers to use those solutions but this similarity can becomean issue if there is no understanding of the differences between the remote andlocal calls. In case of the lack of this understanding the implemented solutionscan be slow and unreliable [10]. The Remote Procedure Invocation can also causeanother problem. Using this technique results in tightening the coupling betweenintegrated applications. Although, they do not share the common data storage(as in the case of File Transfer or Shared Database) the methods called remotelycannot be changed without affecting the integrated applications, i.e. the num-ber of parameters or the return type of the remote method cannot be changedwithout modifying the systems calling those methods.

2.9.4 Messaging

The Messaging (Figure 2.4) is considered as the integration style involving thesmallest amount of assumptions about other parties and hence the most promisingfor performing well in the integration task. Despite that fact, this is the mostsophisticated technique that can be used to solve the integration problem. It canbe said that this approach combines the features of the previous styles. Just likethe File Transfer it allows the applications to be loosely coupled (sent messagescan be transformed in order to comply with the format expected by the receiver,without the sender and the receiver being aware of the transformation itself),but it is also free of its weakness, i.e. high frequency of changes does not causedesynchronisation of the integrated applications and processing of stale data byone of them.


The Messaging enables quicker data exchange and collaboration between inte-grated applications. In contrary to shared database approach it does not coupleapplications to one database. The Shared Database also does not handle wellwith very frequent data changes, especially if the data is being shared betweenapplications placed in different locations, while Messaging is free of this problem.The usage of Remote Procedure Invocations forces to make many assumptionsabout applications and as a result couples them tightly. What is more, thesemantics and syntax of those invocations can be misleading, i.e. causing thedeveloper to think about remote invocations in the same way as he/she thinksabout local invocations. That way of thinking may lead to slow and ineffectivesolutions. Messaging gives the means to transfer data in a quick and efficient way(large number of small data units), with the receiver application being notifiedautomatically if there is another data waiting for the processing.

Messaging also provides a retry mechanism in order to assure the delivery ofthe sent data. Applications integrated using this technique have no need to usethe same unified data structure and are not forced to make so many assumptionsabout each other as in the case of the Remote Procedure Invocation. Messagingalso offers asynchronous data transfer, which means that the sender does nothave to wait for the results in order to continue its processing. It also does notrequire both systems to be operational in order to pass data from the sender tothe receiver. More about the asynchronous method of communication can befound in one of the previous sections called ”Asynchronicity” (2.8).

Chapter 3

Messaging based systems

Messaging was one of the integration techniques that had been briefly describedin the previous chapter. In the current chapter this description will be broad-ened and detailed. As mentioned before, messaging is the most sophisticatedintegration style that in exchange of high complication provides high decoupling,asynchronous communication between integrated applications and other featuresthat make this solution the most flexible among all the described in the previouschapter.

The previous chapter covered messaging in comparison to the remaining threeother integration techniques. This chapter will describe the concept of messaging,key terms connected with this topic and the mechanism by which the messagingbased solutions work. Before going deeper into the description of this techniquean understanding of the basic messaging terms and concepts — such as chan-nel, message, routing, transformation, endpoint, synchronous and asynchronouscommunication — should be perceived.

3.1 Message

In order to transmit the data, it first must be marshaled by the sender into a byteform and then unmarshaled by the receiver so that the receiver has its own localcopy of it. During the transmission data is being wrapped into a Message (Fig-ure 3.1). Each Message forms an undividable entity, it cannot be split into partsor divided. It is the data record that can be transmitted and read by the mes-saging system. In order to communicate the sender’s application must transformdata that is being transmitted into one or more messages and then send thosemessages to the receiver. The receiver gathers these messages, extracts the datafrom them, merges them if the data have been split into more than one Message,and finally processes it. Messaging solutions guarantee delivery of the messageto the receiver (it can be repeatedly transmitted from the sender to the receiveruntil the transmission will succeed).

A message is the smallest undividable portion of data exchanged between

29

30 CHAPTER 3. MESSAGING BASED SYSTEMS

Figure 3.1: Design Pattern Message(source: Enterprise Integration Patterns [7])

integrated applications. It consists of two parts:

• Header — contains information used by the messaging system to describethe data being transmitted, information about the sender, receiver and soon

• Body — contains the data being transmitted, usually this part of themessage is treated as a black box by the messaging systems and sent betweenthe sender and the receiver as it is

Moreover, the message payload might contain special, separated section calledProperties, which contain a list of key-value pairs, defined by the sender ofa message.

The messaging system does not differentiate types of messages being sent.The programmer can choose among different types of messages that can be sent.Those types are as follows:

• Command Message — used to invoke a procedure on the receiver’s ma-chine

• Document Message — used to pass set of data to the receiver’s machine

• Event Message — used to notify the receiver about some event that hasoccurred on the sender’s machine

• Response-Reply Message — used to send a message, which requiresa response from the sender

• Message Sequence — used to send data using multiple messages

The concept of sending a stream of data divided into discrete parts is notonly used in messaging systems. It is also applied in the network protocols,where data is grouped into discrete units of data, i.e. datagrams/packets in caseof the Internet Protocol (IP) and segments in case of the Transmission ControlProtocol (TCP).

CHAPTER 3. MESSAGING BASED SYSTEMS 31

3.2 Message Implementations

The concept of a Message is used in different implementations of integrationsolutions: Java JMS, .NET Messaging, SOAP. Here are some brief informationabout each of those solutions:

1. JAVA JMS

• Message is represented by the class Message.

• Message consists of header, properties and body.

• There are different types of messages depending on the message body,header remains the same for all types:

– Text MessageMessage body contains the String object, which might the contentof a text or XML file or just a text. This is the most commontype of the message. To get the message content the methodtextMessage.getText() is provided.

– Bytes MessageMessage body contains a simple array of bytes. This is the sim-plest and the most universal message type. To get the messagecontent the method bytesMessage.getBytes(array) is provided,it copies the content of the message to array of bytes passed asan argument.

– Object MessageMessage body contains a Java object that implementsjava.io.Serializable interface, so that it can be marshalled andunmarshalled. Method objectMessage.getObject() returns theserializable object containing message’s data.

– Stream MessageMessage body contains the stream of Java primitives. In order toread data from the message body methods such as readBoolean(),readChar(), readDouble(), etc. are provided.

– Map MessageMessage body contains a list of key-value pairs, just likejava.util.Map with String objects used as keys. To get the value ofsome key the method getTYPE(KEY) is provided, where TYPE isvalue’s type and KEY is the name of the key,e.g. getBolean("isEnabled"), getInt("numberOfItems").

2. .NET Messaging

• Message is represented by the class Message.

• Message class has the following properties:

– Body — contains an Object that represents the message content

– BodyStream — stores a content of the message as a stream


Figure 3.2: Design Pattern Channel(source: Enterprise Integration Patterns [7])

– BodyType — specifies the type of data being sent in the messagebody (string, date, number, currency, etc.)

3. SOAP

• Message is represented as an XML document, it contains an optionalheader and a required body.

• An XML document is an atomic data record that can be transmitted.

• SOAP messages can be transmitted using messaging systems (in thatcase the message contains SAOP message as its body).

3.3 Message Channel

The Message Channel (Figure 3.2) is used to transmit Messages between appli-cations. It is a logical address of the message destination in a messaging system.It can be imagined as a pipe connecting sender and receiver inside which messagesflow.

Channels must be defined by the integration team and added to the messag-ing system. Newly deployed messaging system does not contain any channels.They must be created so that applications can communicate using them.

Message channels have very useful functionality — if the receiver is currentlynot available, a message channel will store messages until the receiver will be upagain and will fetch them from the message channel. This feature eliminates theneed for both sender and receiver systems to be on-line in order to communicate.Sending application knows what kind of information it is sending and basing onthis knowledge, it also knows what channel this information should be sent to.It does not have to know which application needs this information. It is sufficientto know that placing the information in a given channel will assure that it willbe delivered to the application that needs it.

The way the channels are implemented varies among different products, butin order to simplify the process of integrating applications, every messaging sys-tem provides an Application Programming Interface (API). The API containsmethods for sending and receiving messages, which hides the details of the com-munication with the messaging system. The application does not have to knowhow the connection to the messaging system is set up, how it is reinitialized in


Figure 3.3: Design Pattern Router(source: Enterprise Integration Patterns [7])

case of an communication error, how the message is being converted into a streamof bytes and so on.

There are two types of channels:

• Point-to-Point Channel — directly connects two applications. Data sentthrough this channel by the sender will be only available to the receiver.Once the receiver will fetch the message from the channel it will be deletedand no longer available.

• Publish-Subscribe Channel — allows sender to send messages throughthe channel to more than one receiver (subscriber). Sender sends data intothe channel, then — independently of the sender — each of the subscribersperiodically checks whether there are any new messages pending in thechannel or, another scenario, the subscriber might also be automaticallynotified about new messages by the messaging system. Then the receiverfetches them from the channel.

The above division is based on the way the messages are being distributedfrom the sender to the receiver. Other division is based on the purpose of themessage channel:

• Datatype Channels — used to avoid confusion when different datatypesare mixed within the same channel. Each type of data being sent hasassigned a different Datatype Channel.

• Invalid Message Channel — used to send error messages to the senderand to provide feedback from the receiver in case of data errors or otherfailures.

• Dead Letter Channel (Dead Message Queue) — used by the messag-ing system when the message cannot be delivered to the receiver.

3.4 Message Routing

In the simplest case of integration solution, systems are connected directly throughMessage Channels. Those connections are straight links from the sender to thereceiver. A Message Channel decouples the sender from the receiver of the mes-sage. Because of this it is possible to have more then one application sending


messages using the same Message Channel. Quite often it is necessary to performsome processing of the sent message before it will be directed to its destination.Messages sent by a single sender may require different processing while beingsent through the Message Channel. Different processing can be required depend-ing on the message origin, business rules, message type or some other criteria.In order to assure this, each Filter component connected to the channel has toknow those rules. However, if the rules change, then all of the components withinthe Message Channel also have to be changed so that they would have updatedrules. This would make any changes to existing solution very time consumingand ineffective, both time and performance like. Very often the components thatwould be used to determine the further processing of the message could not bechanged because it would be too expensive, time consuming or even impossible.

Moreover, in order to determine the further processing of the message(e.g. state if the message is destined for this component or not using businessrules based on the message content) the component has to fetch the messagefrom the Message Channel. But after the message has been consumed, it cannotjust be put back to the channel the same as it was before, because the messagingsystem does not enable that.

In order to solve the problem of redirecting, the message depending on a setof conditions without involving all components participating in the message pro-cessing a new type of component has been introduced into messaging solutions.This component is called a Message Router (Figure 3.3). The role of a routeris to decide where the particular message should be delivered basing on a set ofdefined business rules.

Other components using messaging system are not aware of the router’s exis-tence, because it does not change message content, it only redirects messages tothe proper channel. If the need to change the decision rules will arise, then onlythe router component has to be changed, other components remain unchanged.

A router is a single point where the decision concerning further message trav-elling path is being made, therefore in case of heavy traffic the routing componentmight become a system bottleneck, but the likelihood of such a situation mightbe significantly decreased by using several parallel routing components or byimproving the hardware used to run the system.

The Message Router needs to know the full list of possible message recipientsalong with rules that govern the routing process. The alternate solution, thatcan be used in case of frequently changing list of final recipients, is to let eachof the recipient to decide whether to fetch the message from the queue or not.This alternative solution can be build by using Publish-Subscribe channels andMessage Filters, it is called reactive filtering, while using a routing component iscalled proactive routing.

There are a few possible variants of a Message Router that can be used inintegration solution:

• Fixed RouterThis is the simplest variant. In this variant the router has one input andone output channel defined. It does not perform routing as such, but isused to decouple systems or pass messages between different integration


Figure 3.4: Design Pattern Translator(source: Enterprise Integration Patterns [7])

solutions. Most often this type of routers are used combined with a MessageTranslator or a Message Adapter in order to pass the message betweendifferent integration solutions or different types of message channels.

• Content-Based RouterThis type of routers use the properties of the message such as, for example,the type of the message or the values of the specified message fields in orderto determine the message destination. It is the most commonly used routertype.

• Context-Based RouterThis type of routers use the information about the surrounding environmentto determine the message destination. Those routers can be used to performload balancing or change the message destination if the original recipientis not responding. Context-Based Routers can be used to increase theflexibility and reliability of the system in case of unexpected errors.

Routers can also be divided into two other groups: stateless and stateful.In the case of the first group, a stateless router only considers the message thatit had just received and makes the routing decision basing on only single —current — message. A Stateless router, on the other hand, in order to determinean incoming message destination also takes also into account previous messages.This feature might be used to remove duplicated messages, for example.

3.5 Message Transformation

The concepts covered so far in this chapter concerned the way the message is beingsent from one system to another. What has been omitted is the fact that in orderto communicate, the content of the message has to be properly understood bythe receiver. Message data has to be properly interpreted and used to performnecessary operations, in order to ensure that it must be delivered to the receiver’sapplication in the correct format. In the ideal solution both sender and receiverwould use the same data format, but this situation is very rare. In most casesboth sender and receiver would have to be modified so that they would use thesame data format.

This approach raises many problems to solve. Which data format should beused, the one used by the sender or the one used by the receiver? What kind ofselection criteria should be used to make that choice? Moreover, making internal


changes to the integrated applications can be very difficult or in some cases evenimpossible. It may also cause some changes to the internal business logic ofthe application, which is an undesired situation, because integrated applicationsshould be unaffected by the integration process as much as possible. Makingsuch changes would also neglect the idea of loose coupling described earlier (1.1).After implementing that kind of change into both applications, they would not beloosely coupled anymore. The change in data format in one of them would haveto be reflected immediately in the other one, otherwise the integration solutionwould not work as it was intended to.

The simplest way to ensure that the data format of the arriving messagewill correspond to the internal data format of the receiver’s application is to usea separate component, which will changethe message body to the appropriateformat. This component is called a Message Translator or a Message Trans-former (Figure 3.4). The usage of this component enables to preserve the loosecoupling between applications. In the case of a change of internal data formatwithin any of the integrated applications only the changes in the component per-forming transformation are necessary, the applications will remain unaffected.This way they do not depend on each other, and changes made in one of themdo not enforce to make changes in the others.

The transformation process itself can take place on many different levels ofdata representation. It may refer to the name of the data fields, data represen-tation in those fields, data structure as a whole (different ways of representingthe data) and so on. Hohpe and Woolf [7] makes a division of different levels ofdata transformation and organises them in a similar form as the ISO/OSI model.This division is presented in the table 3.1.

As it is being presented in the above table, the levels of transformations aredivided into four layers:

• TransportTransformations performed in the scope of the communication protocols(Transport Layer) enables data transfer between systems using differentcommunication protocols and ensures reliable message transfer betweenthose systems.

• Data RepresentationThe Data Representation layer performs the transformation concerning therepresentation of the data. Transformation within Transport layer operateson the stream of bytes, while the Data Representation transformation op-erates on the data representation (e.g. it changes the XML representationinto name-value representation).

• Data TypesThe Data Type layer performs the conversion of the data contained inthe message. The conversions includes changing field names, changing datatypes of those fields, combining data from multiple fields into one or splittingdata from one field into two and so on. The goal of this transformation isto make data comply with the data model of the receiver’s application.


Layer Deals With TransformationNeeds (Example)

Tools/Techniques

DataStructures(Applica-tionLayer)

Entities,associations,cardinality

Condensemany-to-manyrelationship intoaggregation

Structuralmappingpatterns, customcode

Data Types Field names, datatypes, valuedomains,constraints, codevalues

Convert ZIP codefrom numeric tostring. ConcatenateFirst Name andLast Name fields tosingle Name field.Replace U.S. statename withtwo-character code.

EAI visualtransformationeditors, XSL,databaselookups, customcode

Data Rep-resentation

Data formats(XML, name-valuepairs, fixed-lengthdata fields, EAIvendor formats,etc.), Charactersets (ASCII,UniCode,EBCDIC), Encryp-tion/compression

Parse datarepresentation andrender in a differentformatDecrypt/encrypt asnecessary

XML parsers,EAIparser/renderertools, customAPIs

Transport Communicationsprotocols: TCP/IPsockets, HTTP,SOAP, JMS,TIBCORendezVous

Move data acrossprotocols withoutaffecting messagecontent.

ChannelAdapter, EAIadapters

Table 3.1: Levels of data transformation

• Data Structures (Application Layer)Finally, transformations within the Application Layer defines the entities,which are used in the application data model and sets the relations betweenthem (e.g. Can customer have multiple bank accounts? Can bank accounthave multiple owners?).

As each transformation can be performed in a separate component, the trans-formations themselves can be chained. Chaining transformations enables to per-form complex changes to the transmitted data on different levels. It also enablesthe designers of an integration solution to combine different transformations intotransformation chains to achieve desired effect in the most efficient way.


Figure 3.5: Design Pattern Endpoint(source: Enterprise Integration Patterns [7])

3.6 Message Endpoints

Up to this point the main massaging concepts concerning the message transporta-tion have been covered. Several topics connected with the message transportationhave been described. Those topics cover issues such as the way the data is beingsent between the applications using the messaging system, what are the basicunits of exchanged data, how does the messaging system cope with sending themto the correct destination system, and finally, how the data is being transformedso that it fits the internal data model of the destination system.

The Messaging system in most cases is a separate application responsible fordealing with all the issues mentioned above, but not only with them. The sys-tems that are supposed to be integrated using the messaging application are alsoseparate business entities. What must be taken into account is that integratedapplications, in most cases, are complex systems, often few or more years old,usually older than the messaging system itself. Quite often, they do not havethe means to communicate with the messaging system — to send messages tothe messaging channels, to fetch incoming messages, to create messages and soon. They also cannot be changed or modified to gain those abilities, because themodification might be too expensive, impossible for technical or some other rea-sons. Thus, in order to make the messaging solution work, one more componentis needed.

Component, which connects the application with the messaging system, iscalled a Message Endpoint (Figure 3.5). It is a custom made component thatperforms all the operations mentioned above and enables integrated applicationto cooperate effectively with the messaging system. It is responsible for creatingmessages, sending them to the message channels, fetching incoming messages,wrapping data from the application into the appropriate message format, andunwrapping incoming messages to extract the data and pass it to the application.

The Message Endpoint encapsulates the messaging system’s Application Pro-gramming Interface (API) from the integrated applications, which are not awareof all the operations being performed when they raise a request to send data toanother application. Because the endpoint is written to work with a particularapplication and messaging system it has to be rewritten if it would have to coop-erate with different applications or messaging system, than it has been designedfor. One instance of the Endpoint can either send or receive messages from thechannel, it cannot do both at the same time. This implies that an applicationcan have several Message Endpoints attached to it.


Figure 3.6: Overview of a communication based on message design patterns(source: Enterprise Integration Patterns [7])

The figure 3.6 summaries all of the terms explained so far. The application onthe left side (Application A) wants to sends a Message to the Application on theright side of the above picture (Application B), the steps of that communicationwould look as follows:

1. Application A sends a Message using its Message Endpoint.

2. The Message is placed in the Message Channel.

3. The Message is directed to the Router, which decides where the messageshould be delivered (let us assume that the message would be delivered aspresented on the picture).

4. The Message is directed to the Translator, which changes the message’scontent format.

5. Finally the Message is delivered to Application B, which using its MessageEndpoint fetches it.

3.7 Synchronous and asynchronous communication

Integration solutions can perform their activities in two possible ways. The term”ways” can be interpreted in this particular case as the overall mechanism bywhich the system operates. Those mechanisms can be of two kinds: synchronousand asynchronous — two ways by which integrated system can communicatewith each other using the messaging system. Those concepts have already beenmentioned earlier in the part covering the integration criteria (2.8), but now wewill try to take a more detailed look at them.

The reliability of communication is a very important issue both in the syn-chronous and asynchronous approach. Some communication errors are beyondthe scope of messaging solution (e.g. physical media error, hardware failure, etc.)but they might prevent messaging system from delivering the message to the


sender. Of course, sender can resend the message after some period of time toincrease the possibility that the receiver will finally receive it. But in the caseof some types of errors this will not assure a successful delivery. This drawbackcan also lead to the loss of application performance and processing speed. Anapplication has to wait until it receives a response instead of performing its usualactivities. In the case of one application accepting requests from several clientsthose drops in performance and processing speed can become even greater andlead to deadlocks.

The main advantage of the synchronous approach is its simplicity. The appli-cation does not have to use additional resources to monitor the whole processingas it is in the case of an asynchronous approach.

This simplified approach is sufficient in case of systems that do not sendsophisticated requests that would require a lot of processing from the messagereceiver. A short processing time of requests will reduce the time the senderapplication has to wait until it receives an answer and as a result, do not affectthe overall performance of the senders in a noticeable way.

Asynchronous communication, in contrary to the synchronous one, is a muchmore complicated concept. The main idea behind it is that application aftersending the request does not wait for a response, but continues its processing.It requires different approach in implementation and design. An application usingan asynchronous model cannot be designed as a sequence of method invocations.It has to be designed in such a way that the remote functionality will be in-voked without affecting the main application flow. This enforces different kindof application design than in the case of synchronous communication. A possiblescenario might look as follows:

1. The sender application (Application A) sends a message (containing mes-sage identifier) designated to the receiver application (Application B).

2. If the receipt of the message has been confirmed by the messaging sys-tem, Application A stores the information about the sent message alongwith identifier assigned to it in the database and switches to perform otheroperations.

3. Application B is notified by the messaging system about a new messageand fetches it.

4. Application B after processing the received message, sends a new messagewith the response for Application A request (the response includes messageidentifier sent in the request).

5. Application A receives a new message with the response for its previouslysent request.

6. Application A looks up the database for the request identified by the mes-sage identifier contained in the received message.

7. Application A after gathering all required information, starts its processing.


This approach despite being more difficult in design and implementation,as it has been stated in the section ”Asynchronicity” of chapter ”Integrationstyles” (2.8), has a lot of advantages over the synchronous one, just to namea few: reliability, efficiency, resistance to communication errors, resistance tohigh-loads.

Both of the presented approaches have their pros and cons. In some cases thesynchronous solution is more advisable — when the processing speed is a priorityand the communication model is simple, with simple request that can be pro-cessed quickly. In other cases — when the application reliability, error resistanceor complex communication model has the priority — the asynchronous model hasthe superiority. Although the asynchronous approach requires a quite differentarchitecture of the application and is more time and resource consuming, it mightprove to be more applicable and result in a higher reliability and flexibility levelof the final solution.

Chapter 4

Design patterns in the applicationintegration

Design pattern is a term that has been adopted from the architecture to softwareengineering [1], which describes a well-known method of solving commonly occur-ring problem. In case of computer science design pattern should not be perceivedas a ready-to-use solution, such as a source code, but only as a template that canbe used in multiple situations. In this paragraph the usage of the design patternsin application integration will be discussed and the term design pattern itself willbe explained in a more detailed way.

Designing and implementing an application can be a very complex and com-plicated task. Applications may vary on the technologies used, working environ-ment, performed task, complexity, and so on. But very often the designers wouldencounter the same problems to solve. The design patterns are a set of provedways of solving those problems. They do not contain a ready solution that canbe put into an application to solve a problem, rather, they are template that canbe used to solve the problem [4].

Design patterns also occur in the messaging systems and application integra-tion solutions. Some of the design patterns described in this section derive fromthe basic concepts of the messaging systems described earlier (3). Each of themis an answer for particular problem, for example, how to connect two applicationswithin the messaging system — by using the Message Channel.

As said before, design patterns are only general templates, not a ready touse solutions. The Message Channel pattern, for example, can be implementedin different ways, as a Datatype Channel, a Point-to-Point Channel, a Publish-Subscribe Channel and so on. Each of those channels performs different task,but all of them are based on the same Message Channel pattern template.

As mentioned before, the Message Channel pattern can be applied in variousways. The simplest usage of this pattern is the Point-to-Point Channel thatconnects two different systems directly. If the aim is to deliver the message tomore than one receiver at the same time, the Publish-Subscribe Channel can beused. This channel has one input (Publisher) channel and many outputs (Message

43

44 CHAPTER 4. DESIGN PATTERNS IN THE APPLICATION INTEGRATION

Endpoints). Each output channel is connected to one system (subscriber). Whena message is published, it is being replicated and sent to each of the outputchannels, where it can be consumed by each of the subscribers. The message canbe consumed only once, so after being consumed by every subscriber the messagewill be removed from the channel and will not be processed again.

Another type of pattern that can be used is a Message Router. The conceptof the Message Router has been already described as this is one of the basicconcepts connected with systems integration (3.4). The Message Router patternis a response to the problem of delivering the message to the correct recipient.Again as in the case of Message Channel, basing on one pattern, different variantsof Message Routers might be built. Those routers variants can be as follows:

• Content-Based RouterThe Content-Based Router (Figure 4.1) reads the message content and bas-ing on it and encoded routing rules directs message to the proper recipient.

Figure 4.1: Design Pattern Content-Based Router(source: Enterprise Integration Patterns [7])

• Message FilterThe Message Filter (Figure 4.2) works in a similar way to the Content-BasedRouter. It reads message content and checks if it matches the encodedcriteria. If it does, it sends the message further, if not the message isdiscarded.

Figure 4.2: Design Pattern Message Filter(source: Enterprise Integration Patterns [7])

• Dynamic RouterThe Dynamic Router (Figure 4.3) is a more flexible variant of the MessageRouter. It allows the routing rules to be modified by sending control mes-sages to the given port of the router. This makes is more flexible then therouter with fixed routing rules and allows to change routing rules dynami-cally, when such a need arises. It can be useful when a new system is being

CHAPTER 4. DESIGN PATTERNS IN THE APPLICATION INTEGRATION 45

Figure 4.3: Design Pattern Dynamic Router(source: Enterprise Integration Patterns [7])

connected to the messaging solution and all routers in the system have tobe updated with new routing rules.

• Recipients ListThe Recipients List (Figure 4.4) extends the functionality of the Content-Based Router. It works in a similar way to the Publish-Subscribe Channel— inspects the incoming message and basing on the message content itdetermines the list of the message recipients, then it forwards the message tothose recipients. The list of recipients may vary depending on the messagecontent, which also can be specified dynamically.

Figure 4.4: Design Pattern Recipients List(source: Enterprise Integration Patterns [7])

• SplitterThe Splitter (Figure 4.5) is used when an incoming message contains mul-tiple elements, which cannot all be processed in the same way. In that casethe message is split into separate elements and each of them is sent inde-pendently to an appropriate system to be processed. The Splitter producesone message for each element contained in the incoming message (e.g. ifan incoming message contains order data with a list of ordered items, foreach item from the list a new message will be produced and published toan appropriate channel).

• AggregatorThe Aggregator (Figure 4.6) works in the opposite way to the Splitter de-scribed above. The Aggregator receives incoming messages and identifiesthe ones that are correlated with each other. When the complete set ofcorrelated messages has been received, it performs an aggregation of thosemessages collects the information from each of the correlated message andpublishes a new — single — message, containing all of the collected infor-mation.


Figure 4.5: Design Pattern Splitter(source: Enterprise Integration Patterns [7])

Figure 4.6: Design Pattern Aggregator(source: Enterprise Integration Patterns [7])

While designing an Aggregator the following things must be set:

1. conditions for correlation (using this conditions the messages will beclassified as correlated with each other)

2. completeness condition (when the set of correlated messages is com-plete)

3. aggregation algorithm (how to process the information from correlatedmessages and publish them into one message).

What is worth mentioning, is that unlike a Content-Based Router, an Ag-gregator is stateful, i.e. it remembers each of the incoming messages untilthe complete set of the correlated messages has been received and processed.A simple Content-Based Router is stateless — it only processes the incom-ing message without any regard to the messages processed earlier (it doesnot keep information about previously processed messages).

• Routing SlipThe Routing Slip (Figure 4.7) allows to determine the whole processingpath for every message. Each incoming message has a routing slip attached

Figure 4.7: Design Pattern Routing Slip(source: Enterprise Integration Patterns [7])


to it, specifying the sequence of the processing steps for this particularmessage. Every processing component is being wrapped in a special routerthat reads the routing slip attached to the incoming message and forwardsthe message to the next processing step from the routing slip. This way,a whole processing chain can be composed and managed from one location.Moreover, the Routing Slip for a new type of messages can be defined ifnecessary.

• Process ManagerThe Process Manager (Figure 4.8) works in a similar way to the RoutingSlip, although it works in a more dynamic way. It forwards the messageto the first processing unit and basing on the processing results from thisunit and the information about the processing step executed previously itdetermines the next processing step. The next step is computed dynami-cally basing on the processing result and information stored by the processmanager. The processing path is not fixed as in the case of the RoutingSlip but is constructed dynamically by the Process Manager.

Figure 4.8: Design Pattern Process Manager(source: Enterprise Integration Patterns [7])

• Message BrokerThe Message Broker (Figure 4.9) is a central component of the integrationsolution. It connects all integrated system. Within its internals it containsdesign patterns described before used to effectively route the messages be-tween the connected systems. It reduces the number of message channelsrequired to connect the integrated system. If each pair of the integratedsystems, which need to interact with each other, would be connected di-rectly, the number of required channels would increase to an unmanageablenumber. The Message Broker significantly reduces the number of the re-quired channels and becomes a central component of the system, where allmessage routing operations are being performed.

As it might be observed there are a lot of routing components that can be cre-ated basing on the message router pattern. Each of them response to a differentkind of need and can be used to solve a particular problem. Those componentsvary from the simplest ones with fixed routing algorithms to the more compli-cated that perform routing dynamically basing on the results of the applicationprocessing and dynamically builds a processing path for an incoming message.

Another pattern that is commonly used in the integration solutions is theMessage Translator. As mentioned before, the Message Translator pattern is


Figure 4.9: Design Pattern Message Broker(source: Enterprise Integration Patterns [7])

used to reformat the data in such a way that it would fit to the internal datarepresentation model of the other system. Such a need may arise very often asthe systems being integrated usually have different internal data representationmodels. In case of this pattern, as well as in the case of the previous ones, thepattern itself is a base for different variants of the Message Translators that canbe used in various situations depending on the faced problem. The idea of theMessage Translator concept has been already described in the section devotedto the main concepts of the messaging systems (3.5). Now let us concentrate onthe description of the different variants of translators based on the same MessageTranslator pattern:

• Envelope WrapperThe Envelope Wrapper (Figure 4.10) wraps sent data into an envelopein such a way that it fits the message format used by the given messagingsystem (adds header and body sections, encryption, etc.). After the messagearrives at its destination point it is unwrapped by the unwrapper, whichwithdraws any modifications done by the wrapper and passes the data, asit was initially sent by the sender application, for further processing.

Figure 4.10: Design Pattern Envelope Wrapper(source: Enterprise Integration Patterns [7])

• Content EnricherThe Content Enricher (Figure 4.11) is used when the destination systemrequires more information than the sender can provide. Content Enricher isable to add additional information to the message fetched from the externaldata source. After this step the message is forwarded to the next processingcomponent.


Figure 4.11: Design Pattern Content Enricher(source: Enterprise Integration Patterns [7])

Figure 4.12: Design Pattern Content Filter(source: Enterprise Integration Patterns [7])

• Content FilterThe Content Filter (Figure 4.12) works in the opposite way to the ContentEnricher. When an incoming message contains complex information andonly a small part of that information is required by the message receivera Content Filter removes the obsolete data from the message, leaving onlydata needed by the message receiver.

• NormalizerThe Normalizer (Figure 4.13) is a combination of the Message Router andmultiple Message Transformers. It is used when integrated systems usedifferent formats of the messages and when each of those formats requiresa different type of translation in order to fit into the model used by the mes-saging system. Information can arrive as an XML document, as a plain textfile containing a comma separated data fields, as an Excel file, and so on.Each of those formats requires different processing in order to transform itinto the format appropriate to the messaging system. When those messagesarrive at the Message Router, they are being forwarded to the appropriateMessage Transformer responsible for dealing with this particular data for-mat. The range of accepted incoming message formats might be easilywiden by adding a new routing rule to the Message Router and connectingthe Router to the additional Message Transformer by the Message Chan-nel. This way the integration solution might be dynamically adapted to thechanging business environment and extend its functionality.

The above examples of integration patterns show how a single template canbe used to solve different kinds of challenges of the same nature, in this case,connecting multiple computer systems.

Single template can become the source for various types of components de-signed to solve different types of problems. Each of those patterns finds an appli-


Figure 4.13: Design Pattern Normalizer(source: Enterprise Integration Patterns [7])

cation in designing an integration solution. Using them helps to overcome some ofthe most commonly encountered problems and makes the designed system morereliable, flexible and easier to support.

Chapter 5

Enterprise Service Bus

Having covered the basic theory of integrating computer systems — styles ofintegration, messaging based systems and design patterns connected with them— let us move forward to the practical usage of the above concepts.

This chapter will describe the integration solution that is currently the mostpopular and most frequently used in the process of integration — the EnterpriseService Bus (ESB). First, definition of this concept will be provided. Later ona Message Oriented Middleware (MOM), which is another integration solutionthat is used as a basis of the ESB concept will be described. After that, theESB aims and capabilities and finally the design patterns connected with thistechnology will be presented.

5.1 Definition

The Enterprise Service Bus (ESB) is an integration solution that enablesintegrating systems in a loose-coupled way. It heavily uses open standards suchas XML and WebServices. It is based on the Message Oriented Middleware,which provides reliable communication using messages. It simplifies creatingcomputer systems architecture focused on providing business services — servicesthat have a meaning to the business, not implementation services, services thathave a meaning to the developers.

Currently there is no formal, industry-agreed upon definition of an EnterpriseService Bus. A lot of vendors provide products claiming they are ESB solutions,but there is no precise definition of what such a product should contain. One ofthe methods of explaining what an Enterprise Service Bus is, is focusing on thecapabilities that it is able to provide and the advantages of deploying it into thecompany. This approach will be taken in this chapter.

According to the Gartner Group an ESB [14] consists of the following fourthings:

• Message Oriented Middleware (MOM)

51

52 CHAPTER 5. ENTERPRISE SERVICE BUS

• Web Services

• Intelligent Routing based on content

• XML data transformation

It is worth keeping in mind that the term ESB does not necessarily have torefer to the software product. It may also beconsidered as:

• a pattern

• an architectural component

• a hardware component (there are devices, which have all the capabilitiesrequired in order to be called as an ESB solution)

One of the following sections named ”ESB components” (5.6) will describethe meaning of an Enterprise Service Bus as an architectural component.

5.2 Message Oriented Middleware

The Message Oriented Middleware (MOM) is the basis of an EnterpriseService Bus. The main task of a MOM is to provide:

• reliable transport

• efficient method of communication using messages

• end-to-end reliability

In a typical Remote Procedure Call (Remote Procedure Invocation) basedcommunication scenario if one of the applications is not available the whole re-quest cannot be performed. The invoking application has to decide what to donext, repeat the attempt within couple of minutes or return an error stating thatthe request cannot be performed. In the case of the message based communi-cation Message Oriented Middleware is responsible for delivering the message tothe proper application. The sender application just puts the message into thequeue and leaves the responsibility of delivering it to the MOM [9]. If the receiverapplication is not running, the message is left in the queue until the receiver willbe up again and will fetch it.

The left side of the figure (Figure 5.1) presents a typical RPC based com-munication scenario, while the right side presents message based communication.Application 1 communicates with Application 2, 3 and 4. Let us assume thatApplication 4 is currently unavailable. There might be dozens of reasons for sucha situation — the network connection might be broken, Application 4 may experi-ence some technical problems or simply it may be down because of a maintenance.Application 1 must get data from all three applications, but Application 4 is un-available. Which action should be taken — repeat the attempt within a coupleof minutes or return an error — this decision must be made by the Application1.

CHAPTER 5. ENTERPRISE SERVICE BUS 53

Figure 5.1: Overview of a communication based on Remote Procedure Calland Message Oriented Middleware

Let us apply the same scenario to the right side of the figure. Application 5communicates with Application 6 and 7. Application 5 does not have to worryabout the applications’ accessibility, because it does not communicate directlywith them — it is using its message queues, which are always available. If Ap-plication 6 or 7 is currently not working, the messages designated for them willnot be dropped, but they will be stored in the queues until the Applicationswill be up again. Message Oriented Middleware guarantees that eventually everymessage will reach its destination application. Application 5 does not have to beconcerned about it.

5.3 Tightly coupled interfaces

To better illustrate the overpowering number of connections in a tightly coupledinterfaces scenario, implemented for example by the Remote Procedure Invoca-tion, let us consider the following situation. When every system provides oneinterface and is connected with every other system, the total number of connec-tions might be computed using the formula [3] n(n−1)

2, where n is the number of

the systems. If there are 5 systems (n = 5), the total number of connectionsis 10. That amount of connections is still manageable, but if there are 10 sys-tems then there are 45 connections and in the case of 100 systems there are 4950connections! Of course, not every system has to be connected with every other,but it shows that the growth of that kind of computer software infrastructure isvery limited.

5.4 ESB aims

One of the aims of introducing an ESB is to decouple the client (applicationneeding an access to the service) from the service provider. The left side


Figure 5.2: Tightly coupled system architecture compared toan Enterprise Service Bus architecture

of the figure (Figure 5.2) shows the typical model of interconnected systems.Component1, in order to connect to Component4, has to know the following:

• Component4 IP address

• Component4 port number

• Component4 protocol and a client of that protocol (EJB)

• Component4 method name and its signature (types of arguments, type ofreturn argument, throws declaration, etc.)

The same applies for all of the connections on the above picture. Before onecomponent will be able to connect to another, it has to know a lot about theother party and also, assume that this information will not change.

The introduction of the ESB, presented on the right side of the figure 5.2,relieves the client from the need to know, who is providing the service, becausethe ESB is responsible for creating a communication channel between the clientand the service providers. That way, the application does not need to havethe integration code, because it will no longer be responsible for creating theTCP/IP connections, reconnecting in case of a communication error, knowingconnection information (URL, port number) of the service provider and so on.Thus, introducing an ESB will simplify the design of the client. Furthermore, ina tightly coupled system, any change of the connection data in one of the serviceproviders, also requires a change in all of the clients that are using this serviceprovider. This is no longer true in the case of an ESB, because it is the ESB,which is responsible for storing that information.

Let us imagine what would happen if the IP address of Component4 wouldbe changed one day? Then in the case of a tightly-coupled scenario all of theapplications connected directly to the Component4 (i.e. Component1 and Com-ponent5) will have to be updated. Now, in the case of an ESB scenario only theconfiguration hold by the Enterprise Service Bus will have to be updated.


To sum it up, an ESB provides service location transparency, as theclient no longer has to know the exact location of the service provider — it is theresponsibility of the ESB.

An ESB also enables sharing of services. Once implemented service mightbe used in many projects or by clients from various departments. For example,service providing information about the employee might be used by the systemsfrom the HR, financial or IT department.

Also, what an ESB enforces is the separation of the business service fromthe implementation service. Companies are run by business rules and theyshould be seen from the point of the business services they provide. Usually, itis the developers, who by creating EJB remote interfaces, WebService interfacesand remote procedure call methods are defining the way, in which the companyis perceived. An Enterprise Service Bus enables to change that. Thanks to beingable to transform business request into an invocation of the particular implemen-tation services it enables the business — not the developers — to define the waythe company is seen by its business partners and customers. The responsibility ofthe business is to know how to run a company — what the company’s input andoutput are and what services to provide, while, the responsibility of an ESB is toknow what the IP address and TCP port of the application server are, what theprotocol used by particular system is, what the signature of a particular methodis, etc. If one day, one particular implementation service will be replaced bya new one, the way the company does business will not change, so that cannotenforce any modifications on the other — business partner or customer — side,only the ESB will have to be updated. This enables to decouple the businessmodel — the way the company works — from the implementation.

5.5 ESB capabilities

This approach — describing an ESB by its capabilities — is common in existingliterature, another example of such list can be found in IBM Redbook concerningthe subject of Enterprise Serivce Bus [12].

1. Message Transformation — ability to transform structure and format ofbusiness request to a service provider

An ESB must be able to change the message format to the format acceptedby the service provider.

2. Message Routing — ability to send a request to a particular service providerbasing on some criteria

An ESB must be able to decide to which service providers a particularmessage should be delivered.

3. Security — ability to protect an ESB from unauthorised access

An ESB has to assure than only selected clients have access to particularservices.


4. Transaction Management — ability to provide single unit of work for abusiness service request by providing a framework for the coordination ofmultiple resources across multiple services

There is no ESB product that provides a transaction management, becausean ESB connects multiple different IT systems and it is impossible for itto rollback a distributed transaction after it has been committed by one ofthe systems.

Let us see an example of the problem mentioned above, the possible scenariomight look as follows:

a) The ESB receives a message, according to a set of rules it knows thatit has to connect first with System A then with System B.

b) The ESB communicates with System A.

c) System A sends some messages into a Message Channel and returnsinformation back to the ESB that it has successfully completed thattask.

d) The ESB communicates with System B, but System B is unable toperform the request because of internal errors, it returns an error.

e) The current situation is that System A has successfully completedthe transaction and System B has failed. The ESB cannot rollbackthe operations (sending messages into a Message Channel) taken bySystem A, it can only notify System A that this transaction should berolled back, but how to handle it is delegated to the System A.

To sum up, no ESB provides transaction management, because it is impos-sible for an ESB to rollback operations performed by one of the integratedsystems. An ESB can only provide a framework for transaction manage-ment.

5. Message Enhancement — ability to modify and add information as requiredby the service provider

It might be necessary for the ESB to add some additional informationinto the message before it will be delivered to the service provider. Thatinformation might come from a database or some other system.

6. Protocol Transformation — ability to accept messages sent using differentprotocols, i.e. IIOP, SOAP/HTTP, CORBA

This capability consists of two aspects:

• logical — an ESB must understand the protocol (its semantic andsyntax)

• physical — an ESB must have a component suited for operating usingthat protocol, i.e. HTTP server, CORBA client, etc.


7. Service Mapping — ability to translate a business service into the corre-sponding service implementation and provide binding and location infor-mation

It is a mapping performed by an ESB between abstract business serviceand implementation service (IP address, port, name of the method, etc.)

8. Message Processing — ability to monitor the state of the received request

For the client, sending a message to the ESB, the most important thingis that a sent message should never be lost. In order to achieve that theESB has to monitor the state of the message — is it already processed bythe service provider, was the processing successful, is the service provideravailable, etc.

9. Process Choreography — ability to manage complex business processes thatrequire the coordination of multiple business services to fulfil a single busi-ness service request

This functionality enables the client to perceive business requests as onesingle request, while in fact its execution may trigger the execution of mul-tiple business services. It is usually implemented as a Business ProcessExecution Language (BPEL), which is a language enabling business pro-cess modelling [9].

10. Service Orchestration — ability to manage the coordination of multipleimplementation services

The difference between previously described process choreography and ser-vice orchestration is the type of service being managed — in case of serviceorchestration it is an implementation service, in case of process choreogra-phy it is a business service.

One of the reasons for not having a formal definition of the ESB is that everyapplication has different requirements for the capabilities provided by an ESB.For example, in some integration projects a transaction manager or a BPELengine might not be required, but there might be a need for message routingand message transformation functionality. There are products on the market,which do not have a support for transactions and BPEL, but do have supportfor message routing and transformation. They claim themselves to be an ESBsolutions and, what is the most important, they would fulfil requirements of suchan integration project.

As it has been depicted the term ESB has a very broad meaning and it isbeing used by the software vendors to describe products providing differentialfunctionality.

5.6 ESB components

An ESB does not have to refer only to a software product, but can also refer toan architecture component. This section will be devoted to this meaning of theterm ESB.


While designing an integration solution an ESB should not be perceived fromthe perspective of one particular software and its capabilities, but rather as a com-ponent, which functionality might be provided by multiple products available onthe market. This approach to an integration does not tie the company deployingan Enterprise Service Bus to one concrete solution, but enables the company tohave products from various vendors cooperating. One of the conceptual modelsof such an ESB architecture is presented on the figure (Figure 5.3). It consists ofthe following logical components:

1. MediatorThe Mediator is the most important component in an ESB. The crucialfunctionality provided by this component is the routing, communicationand protocol transformation. A product not having the above cannot beconsidered as an ESB solution. The Mediator is used as an entry point forthe ESB — messages sent to the ESB are received and processed by thiscomponent. It might also be responsible for message transformation andenhancement. In order to enable reliable and secure processing of requestsit must support security, error handling and transaction management.

2. Service registryService registry is a component that provides the functionality of the servicemapping.

3. ChoreographerThe role of the choreographer is to enable process choreography — co-ordination of business processes. This component is actually a client ofan ESB. It has the knowledge about the sequence of business services thatmust be called in order to perform one — sophisticated — business request.If the mediator decides (according to its rules) that this particular requestneeds to be choreographed, it will be forwarded it to this component. TheChoreographer after looking up its configuration will invoke proper serviceproviders by sending messages — just like an ESB client — to the mediator.

4. Rules engineThe Rules engine is an additional component, which may not be requiredin some integration projects. This component enables to have a rule-basedrouting. Its functionality includes: message routing, message transforma-tion, security and transaction management.

5.7 Open source ESB products

Among dozens of ESB products available of the market there are two open sourcesolutions, each with different characteristics:

• The Mule is a Lightweight Messaging Middleware Framework. It intro-duces the concept of Universal Messaging Objects (UMO) — just PlainOld Java Objects, which are responsible for communicating with the service


Figure 5.3: Conceptual overview of an Enterprise Service Bus components

providers, performing transformation and routing of the messages. They aredeployed in a mule container — a framework enabling the communicationbetween endpoints.

• The ServiceMix is a JBI-compliant ESB. It consists of the Normalized Mes-sage Router with a bunch of components responsible for protocol transfor-mations and ESB capabilities (rules engine, etc.)

5.8 ESB integration patterns

5.8.1 VETO pattern

Figure 5.4: ESB Design Pattern VETO

VETO (Figure 5.4) stands for Validate, Enrich, Transform, Operate. It isa widely used integration pattern in ESB solutions. The VETO pattern [3]ensures that data exchanged in an ESB will be consistent and valid. Each com-ponent in the VETO pattern might be implemented as a separate service, whichmight be configured and modified independently of any other component.


Validate

The aim of the validate step is to ensure that messages received by the serviceprovider will have proper syntax and semantics. This step should be performedindependently — not inside the service provider because that solution wouldlimit the re-usability of validation and complicate any further modifications of it.Moreover, implementing validation as a separate component would ensure thatevery message that gets to the service will be in a proper format, thus wouldsimplify the design of service provider and enable the Operate step to focus onbusiness logic. The simplest way of validating an incoming message is to checkwhether the message is a well-formed XML document and conforms to the XMLschema or WSDL, but there are also other possibilities, like for example validationscripts.

Enrich

The aim of the enrich step is to add some additional data to the message contentthat would be needed by the service provider, for example, information aboutthe customer, who has placed order. That information might be fetched from thedatabase or might be the result of invoking another service.

Transform

The aim of the transform step is to change the message format to the one acceptedby the service provider. This step might transform the message into an internalmessage format of the service provider, releasing the Operate step from the needto perform this task and therefore increasing its efficiency.

Operate

The aim of the operate step is to invoke target service or to interact in some waywith the target application.

5.8.2 VETOR pattern

The VETOR pattern [3] is a VETO pattern which introduces a new componentplaced right before the Operate step — the Router. The aim of this step is todecide whether a message should be delivered to the service or not. The routermight be implemented as a part of the transform component or, more preferably,as a separate service.

5.8.3 Two-step XRef pattern

In an ESB two types of transformations are taking place: structure and contenttransformations. The aim of the structure transformation is to change the formatof a message, while the aim of the content transformation is to enrich a messagewith some additional data, usually fetched from the database. This process is


usually performed in one step, when the output from one step is used as an inputto the following step:

• XSLT transformation

• XPath query

• JDBC query

• SQL statement

The concept of the Two-step XRef pattern [3] (Figure 5.5) is to create twoseparated components responsible for only one type of operations:

• XML parsing: XSLT transformation and XPath query

• Database lookup: JDBC query and SQL statement

This approach has a lot of advantages over the previous model:

• a better code re-usability: components for XML parsing and databaselookup might be used in multiple different projects

• an easier and quicker development: both components might be developedsimultaneously by different teams

• loose coupling: problems with the database does not affect the operationof the XML parsing component

Figure 5.5: Comparison of the internals of a typical transformerand Two-step XRef transformer


5.8.4 Forward Cache Integration pattern

After introducing the ESB into a company, the need of having a single application,which will gather the information from all of the integrated systems and presentthem in a coherent visible form, quite often arises. Portal applications performsuch a role. They enable pulling data from multiple sources like other systems anddatabases and representing them in a unified way through web pages. There arevery useful, because the user does no longer have to seek for data in many systems,which might provide various user interfaces (web pages, command line, etc.),might require different credentials and so on. All the information is available inone place.

Despite mentioned advantages of the portal applications they also invite newchallenges. In order to present data on the web page the portal must be able toget it within a couple of seconds. For a lot of integrated systems it might notbe possible to fulfil that requirement. There might be applications designed asterminal applications, used by a single user, which are not adjusted to handlethe throughput required by the portal application. Moreover, systems accessedby the portal application must be available 24/7. For some, which were designedas desktop programs run on a personal computer, this requirement might notbe fulfilled as they might require periodical restarts and are not resistant tothe hardware problems of the computer on which they are running. Because ofgeographical separation data from the integrated systems is available only whenthere is a network connection between them and the portal system. When thenetwork connection is broken, the portal application will not be able to presentany data to the user. Thus, it becomes crucial to have a properly working networkinfrastructure.

Therefore, it is very important to keep in mind that systems which are aboutto be integrated will face a lot of new challenges and problems, which might nothave been considered by the authors of them.

One of the ESB services, which might help with solving some of those problemsis the Cache Service. The task of this service is to store the results of serviceinvocations returned from service providers. This service combined with the For-ward Cache Integration pattern [3] enables the portal application to accessthe data directly from the Cache Service, once it has been already presented,even if the system, which supplied that data, will be temporary unavailable.

There are two possible scenarios to implement this solution:

• using the publish-and-subscribe model — constantly inform the Cache Ser-vice about the changes in data

• using message routing — duplicate every response from the integrated sys-tem to the Cache Service

Publish-and-subscribe model

In this scenario every change of the data held by an integrated system will causesending a message with a set of changes to a message topic. The Cache Servicewill be a subscriber of that topic. This solution is only suitable for small computer


software infrastructures with systems not frequently changing data. It is not hardto imagine what would happen if there would be multiple integrated systemsconstantly changing their own data, then most of the traffic would be consumedby update-messages making an ESB incapable of handling any regular-messages.

Message routing

This scenario assumes the usage of one of the ESB main components the router.Every response, before getting back to a portal application, should also be sentto the Cache Service. In that way, it will have a copy of every informationthat used to be presented by the portal application and in case of inaccessibilityof an integrated application that information might be supplied by the CacheService.

Chapter 6

Case study: Messaging systemswork principles

Having covered the basic concepts required to get a better understanding ofthe application integration topic, these concepts will be gathered together inform of a case study that would show how to use them when solving a businessproblem. The topic of this case study will be an imaginary integration task ofan international company producing toys. It has been created by the authorsof this thesis in order to illustrate the theoretical background, presented so far,from a more practical point of view.

This company has multiple systems located at distant geographical locations.The company headquarters are located in Europe, along with storage facilities.The factories producing toys are located in Asia. The Logistic System is di-vided into two parts: one responsible for delivering products from factories tothe storage facilities and a second one responsible for delivering ordered productsto the final costumers. The company also has an Internet shop, established inorder to widen the potential number of customers and make its products availableworldwide.

The computer systems infrastructure is comprised of the following:

• Internet Shop — responsible for the interaction with the user and placing(and confirming) orders

• Orders Fulfilment System — responsible for fulfiling the orders

• Storage System — responsible for providing information about productsupplies

• Pricing System — manages the prices of all products available for pur-chase

• Loyalty System — responsible for storing information about discountsfor those customers, who purchase most frequently and/or purchase largequantities of products

65

66 CHAPTER 6. CASE STUDY: MESSAGING SYSTEMS WORK PRINCIPLES

• Payment Check System — responsible for checking the status of thepayments for ordered products

• Supply System — responsible for sending orders to the factory in casethe supplies of a given product would run low

• Logistics System — responsible for delivering packages with orderedproducts to the customers

• Postponed Orders Fulfilment System — responsible for fulfiling ordersthat could not have been finalised earlier because of the lack of products atthe storage

The task at hand is to connect all those system into one big business entityby using the messaging system. The high level design view on the schema of thesystems after the integration is presented on the figure 6.1.

Of course, each of the systems has to have a number of endpoints attached,so it would be capable of sending and receiving messages. Message Channelsbetween the systems must also be set so that the communication can take place.The connections between the systems must be determined earlier (there is nopoint in connecting two systems using a Message Channel if no communicationbetween them will ever occur). During this process a possible location for placingadditional components, such as Message Routers or Message Transformers, shouldalso be determined.

For every router the set of rules must be set by which the router will determinethe destination point of the incoming message. If there is a Message Transformer,the rules for message transformation must also be set.

It also must be decided whether this solution would use a synchronous or anasynchronous communication model. The system being the subject of this studyshould be resistant to the communication failures and as flexible as possible.Also we do not want the Supply system to stop working and receiving requestsfrom the Storage System, if it would not get the acknowledgement of the receivedorder from the factory, and so on. It this case the most suitable model of com-munication would be the asynchronous one. The usage of this model also meansthat much more effort must be put into the design and implementation of thesolution, but it will assure that the final solution will operate in the desired way.

The detailed description of the design and implementation of this case study,covering all possible issues, could easily cover the whole volume of this thesis.Because our goal here is to only give a brief taste of the integration task andshow the practical usage of the concepts described previously, we will just givethe brief description of the integration solution, not focus on the technical details.

First, let us take a closer look at the simplest scenario, involving an order madeby the Internet Shop with credit card payment, when there is no need to orderthe products from the factory and wait for the fulfilment of the order becausethe products are available in the desired quantity at the store. This scenario ispresented on the sequence diagram — figure 6.2.

The user visits the web site of the Internet Store, logs in using his/her username and password, browses through the list of available products, selects the

CHAPTER 6. CASE STUDY: MESSAGING SYSTEMS WORK PRINCIPLES 67

Figure 6.1: Case study systems

one that he/she is interested in and places an order. After the order is beingplaced, the Internet Shop sends a request to the Storage System to check if theselected products are available at the moment. As mentioned before, in ourscenario we assume that the ordered products are available. In the oppositecase, the Storage System would send a request to order them to the SupplySystem. Then the Supply System, after the amount of products needed wouldreach a specified quantity, sends a request to the factory to make those goodsproduced and delivered to the storage in Europe and then forwards the order to


Figure 6.2: Case study processing sequence

the Postponed Orders Fulfilment System.When the Storage System verifies that the requested products are available,

it sends back the response to the Internet Shop application. The Internet Shopalso sends a request to the Pricing System to check the prices of the products.The Pricing system sends a request to the Loyalty System, which checks if theuser, placing the order, has any discounts. After receiving the response it calcu-lates the prices and sends the response back to the Internet Shop.

Having gathered all data the Internet shop application displays informationabout the order along with calculated prices. When the user confirms the place-ment of the order, it is passed to the Order Fulfilment System, which managesthe fulfilment of placed orders. Depending on the selected payment method thesystem sends a message to the Payment Check System in order to check if thepayment has been made, and the order can be processed further (in the case ofa money transfer, payment by credit card) or send request directly to the StorageSystem (in case of payment upon delivery) to prepare the package containing or-dered product (the same request is being sent when the message is received fromthe Payment Check System in case of the payment methods mentioned above).

When the Order Fulfilment System receives a message from the Storage Sys-tem that the package is ready for a shipment it sends a message to the user thatthe order has been completed and is ready for shipment (using e-mail address


of the user). The message is also being sent to the Logistics System to add thepackage to the list of packages to be picked up and delivered on the next day.When the package is sent out to the client the message is being sent to the OrderFulfilment System to notify that the package has been sent, upon receiving thismessage the system sends an notification e-mail to the user informing that thepackage has been sent and stating the approximate delivery time. Sending thisnotification e-mail finishes the order processing by the system.

As it can be seen, even in the case of the simplest scenario the interactionbetween the involved systems is quite complex. There are many systems involvedin the process of exchanging different types of information. It is worth keeping inmind that the data being sent is the subject of many changes and prepossessingbefore it can be consumed by the next system in the processing chain (differentdata formats, internal data model of the applications, and so on).

Designing the integration solution for this system would require the usage ofall of the components explained in the previous chapters, i.e. Message Routers,Message Translators, Message Endpoints. The Message Routers could be usedto determine the destination of a message in the case when one system can sendmessages to different receivers. The Message Translators could be used to trans-late data contained in those messages so that they would fit the internal datamodel of the receiver system.

The schema of the systems integrated by the messaging system and incorpo-rating the elements mentioned above is presented on the figure 6.3.

Although the diagram 6.3 may look simple and straightforward, it is, in fact,an example of a badly — tightly coupled — designed computer software infras-tructure. One of the disadvantages of this solution is the unmanageable numberof message channels (depicted on the figure as arrows). Although, the majordisadvantage is the fact that an application, in order to communicate with otherapplications, must know a lot of details about it, like for example message channeladdresses and message formats. Moreover, every time a message format changesin one application, all applications communicating with that application alsohave to be updated. For example, if the Storage System will change the formatof the date field, applications such as the Logistics System and the Order Fulfil-ment System also would have to change the format of the messages that they aresending to the Storage System.

The solution of the previously mentioned problems might be the usage of inte-gration patterns — the Message Router (3.4) and the Message Transformer (3.5).Figure 6.4 presents a new architecture utilising those concepts. Despite the factthat the amount of message channels has increased, this solution enables greaterdecoupling of the applications. The knowledge about the format of messagesaccepted by the system is now not hard-coded inside each application, but is del-egated to a new, intermediary component — the Message Transformer (depictedon the figure as the letter T). Also decisions about the routing of messages arenot taken by each application, but by the Message Router (depicted on the figureas the letter R). This approach introduces a greater level of loosely coupleness —the message format of each application might be changed independently of theothers.

Each system shown on the figure 6.4 has been enriched by the endpoints


Figure 6.3: Case study: Message Channel scenario

that enable the communication between the system and the messaging system.The number of endpoints corresponds to the number of channels from whichthe given system can receive messages or to which it can send them. MessageRouters are used to direct the messages to their destinations basing on givenbusiness rules (e.g. name of the destination system placed within the messageheader). The Router connected to the Internet Shop channel decides whether tosend a message to the Storage System (checks the availability of the product) orto the Pricing System (gets the prices for a given product) or to the Orders Fulfil-ment System, which passes it (a message containing information about a placedorder) on for further fulfilment. The router connected to the Order FulfilmentSystem channel routes messages either to the Storage System (a message con-taining a request to create the package for shipment) or to the Payment System(checks whether the payment for the ordered products has been made) or to theInternet Shop (notification messages about the various stages of order fulfilment).Message Transformers are used to transform Messages to the format readable by


Figure 6.4: Case study: Message Channel with Router and Translator scenario

the recipient system (e.g. Messages sent from the Storage System to the Inter-net Shop need to be transformed by the Transformer component in order to becorrectly read by the destination system, in particular in the case of the InternetShop). Let us assume that the Order Fulfilment System stores information in sucha way that the receivers name and surname are combined together in the fieldreceiver and the address is stored in the field package destination. Thus, the roleof the transformer would be to extract the receivers name and surname and putthem in the field receiver and then extract the information about the destinationaddress and put it in the field package destination. Only after those transforma-tions the message can be sent further to the Order Fulfilment System. After thosechanges it can be assured the data will be read and interpreted correctly by thesystem and will not cause any errors during its processing.

The problem of the unmanageable number of message channels is resolved bythe introduction of an Enterprise Serivce Bus (Figure 6.5). In this scenario, eachof the systems communicates only with the ESB, which is responsible for mes-


Figure 6.5: Case study: Enterprise Service Bus scenario

sage transformation and routing. This approach simplifies the development andmanagement of applications, because there is only one message channel betweenthe application and the ESB. Moreover, an Enterprise Service Bus provides widerange of adapters — components for accessing an ESB, which removes the needfor an application to know details of communication with the ESB.

An Enterprise Service Bus incorporates the implementation of multiple inte-gration design patterns, such as the Message Channel (3.3), the MessageRouter (3.4), the Message Transformer (3.5), the Message Endpoint (3.6), etc.But the internals of that implementation are hidden from the applications usingan ESB. This simplifies their design and makes the process of integration easier.

The example described above should give a good overview on the usage andpractical application of the concepts presented in the previous chapters. Using themost basic components such as a Message Router and a Message Translator,a complex integration solution might be designed.

Chapter 7

Implementation

This chapter will cover the details of the implementation of the ESB platformnamed pESB. It was developed by the authors of this paper as an internal partof this master thesis.

First, the basic concept behind this product will be described, its technologyand architecture. Later, the internal working of it, illustrated by the sequencediagram, will be presented. Having covered the essential information, a real lifeexample, which will familiarise the reader with the process of configuring theESB, along with source code snippets, will be introduced. Finally, the problemsthat occurred during the implementation will be presented.

7.1 The origin of the name

pESB is an acronym for Polish-Japanese Institute of Information TechnologyEnterprise Service Bus. Because the authors of pESB belong to a group of busypeople that do not want to waste time typing long names, the name pESB willbe used throughout this chapter.

7.2 Concept

Being aware of our limited amount of time and available resources we decided tocreate an integration solution that will be lightweight and easy to use in the firstplace. We knew that we would not be able to create a tool that will competewith integration products existing on the market for a couple of years. Thus, wedecided to create an integration product that will provide simple functionality,but will have an architecture that will enable easy scalability and further devel-opment.

The main concept of this approach to an ESB solution was to base implemen-tation on the standards and technology already available on the market, like JavaEnterprise Edition (EJB, JMS), XML, WebServices, etc. This approach is verydifferent compared to other existing ESB solutions. pESB takes advantage of

73

74 CHAPTER 7. IMPLEMENTATION

the services provided by the application server like security management, trans-action management, pool and resource management, etc. It does not come withits own Message Queue implementation. In our opinion this should be treatedas an advantage, because we do not force users to use any particular solutions.Nowadays, every company thinking about integration solutions has already an in-frastructure that might be used. Furthermore, our solution does not reinvent thewheel. The products providing those features have been available on the marketfor a long time, and certainly the authors of those products had to overcomea lot of problems and solve a lot of issues that appeared during the usage of theirproducts by the customers.

pESB was designed as a multi-tier application (Figure 7.1). The basis of thissolution is the Message Bus. Its main responsibility is to provide reliable methodof communication between multiple points using data channels.

The next layer is the Application server providing a lot of facilities, whichan application running in a container can take advantage of. Mentioned facilitiesare as follows:

• Transaction management — container is responsible for starting transac-tions, committing, and automatic rollback in case of an error

• Security management — authentication and authorisation done by the con-tainer

• Pool management — the container is also responsible for managing poolsof Enterprise Java Beans instances, pools of connections to the resources(allocating new, removing not used)

• Resource management — the container provides an application with theconnectivity to the resource like databases or JMS queues

• Multi-threading — the container is responsible for starting threads andmonitoring their work

pESB as an application running in the container might take advantage of allthose facilities mentioned above.

7.3 Technology

pESB is an enterprise application using the set of state-of-the-art Java technolo-gies:

• Java 6 — runtime Java source code compilation

• EJB 3.0 — no XML descriptors, the whole configuration written in anno-tations

pESB has been implemented on Glassfish V2 application server, but sincethere are no XML descriptors specific to the chosen application server portingpESB to any other EJB3-compliant application server would not require mucheffort.

CHAPTER 7. IMPLEMENTATION 75

Figure 7.1: Integration architecture layers

7.4 Classification

According to the classification presented in the Forrester Wave comprehensive re-port about Enterprise Service Bus products available on the market[13] customerrequirements might be yielded into two segments:

• ”keep it simple”

• ”I want it all now”

For the first group of customers the most important thing is that the solu-tion should be simple as possible in order to enable low-cost integration. Also, itshould have a plug-in architecture to enable easy customisation to customer’sneeds. The products from the second group, on the other hand, should featurewide range of additional services like Business process management, process sim-ulation, monitoring or optimisation. pESB will suit customers from the firstsegment.


7.5 Architecture

The main focus in the development of pESB has been put not on the optimi-sation of particular methods or functionalities but on having an architecture,which will enable scalability, distribution, reliability and security of the solution.One of the ways to achieve that goal was to use asynchronous processing oversynchronous. The internal communication between components of the pESB isdone using asynchronous messages (Figure 7.2). The message processed by onecomponent is put into the queue of the other component. Every message is per-sisted. The combination of those two factors provides scalability and reliability.Message queues enable to have multiple consumers running on different com-puters. Thus, the number of consumers may change dynamically, new con-sumers might be added and removed on the fly. This feature provides scalability,pESB might be easily adjusted to the growing requirements of processing greateramount of the messages, simply by adding new application servers.

Figure 7.2: pESB architecture

The overall process of communication is depicted on figure (Figure 7.3).The client communicates with the ESB using an Agent, which is a software com-


ponent providing an API to access the pESB. From the client’s point of view(depicted on the figure as System1 and System2) the whole complexity of mes-sage based communication, asynchronous processing, XML documents, etc. ishidden. pESB except providing an interface for receiving and fetching messagesalso provides a method of dynamic configuration (interface Config) using JavaAPI.

Figure 7.3: pESB communication architecture

7.6 Processing sequence

In the presented processing sequence (Figure 7.4) there are three participants:

• a Sender system, named System1

• the pESB

• a Receiver system, named System2

System1 is communicating with System2. Both of the systems are usingagents to communicate with the integration platform. The agent is a softwarecomponent responsible for the whole communication with the pESB. The detailsof that communication, whether it is done using EJB or SOAP/HTTP is hiddenfrom the client (System1 and System2). Moreover, to simplify the design ofSystem1 and System2 clients they operate on Data Transfer Objects (DTO), noton XML documents.

These are the actions that happen when System1 sends a message to thepESB and when System2 is the set as a receiver of that message:


Figure 7.4: pESB processing sequence

1. System1 invokes the send method from the Agent passing as an argumentData Transfer Object, which it would like to send to the ESB.

2. System1’s Agent receives the DTO and creates an XML message out of theDTO content, then invokes the method from the remote interface of pESBusing EJB and passes that XML document as an argument.

3. The ReceiverBean in pESB receives that XML document and puts it intoan input queue and returns information back to the Agent about the statusof that operation.

4. After some time...

5. The Message-Driven Bean named TransformerBean in pESB fetches mes-sage from the queue, asks the Transform and routing engine to performtransformation and routing of that message, then:

• If the message is already processed by pESB and is about to the routedto the destination system, sends it to the output queue.

• If not, sends it once again to the input queue.

6. After some time...

7. System2 wants to fetch new messages, it invokes the receive method fromthe Agent.

8. System2’s Agent invokes the method from the remote interface of pESBusing EJB.


9. The FetcherBean in pESB checks whether there are any new messages wait-ing in the output queue for System2, if there is a new message, fetches itand returns back to the System2’s Agent.

10. System2’s Agent creates a new Data Transformer Object using the contentfrom the XML document and returns that new object back to System2.

It is worth mentioning that all the operations performed inside the Transformrouting unit are performed in a transaction. This means that in case of an errorno message will ever be dropped.

7.7 Configuration

All the configuration is done in the runtime, no restart or reload is requiredafter changing the configuration, also no XML file editing is required. The wholeconfiguration might be changed using a Java API. Furthermore, even a Java codeprovided by the user is compiled on the fly and functionality provided by it isavailable right away.

Each component in the pESB — router or transformer — might be configuredin multiple ways. That is, the transformer might be configured using either anXSLT file or a Java code. It is up to the user which method of configuration he/shewill choose. In the case of a Java code method, the source code provided by theuser is compiled on the fly, during the process of configuring. This approachhas a lot of advantages over dynamic execution of the code. The compilationprocess can detect a lot of errors, which in the case of a dynamic execution wouldbe found out only at the runtime. Moreover, the user in order to develop acomponent might use an IDE of his own — Eclipse, Netbeans or IDEA, etc.,which gives him/her the possibility to avoid common programming mistakes likemisspelling variable names, methods, using wrong types and so on.

The process of configuration pESB is performed using a Java API. We believethat this solution provides the greatest flexibility, because it does not enforce theway the configuration data will be stored. In this approach, it is possible to savethe configuration in an XML file, a LDAP directory, a relational database, etc.The only thing that must be done is the creation of the import tool, which willread the data from the particular medium (XML, LDAP, database), parse it andinvoke proper methods from the Java API. Moreover, this approach also enablesto have some addition logic in the import tool itself, which will be called beforeusing the configuration Java API. Furthermore, it is easier to modify an importtool (XML or database) than a system (pESB).

7.7.1 Configuration model

The model of pESB configuration (Figure 7.5) incorporates two components:

• System

• Transform Routing Unit (TRU)


The System represents a system taking part in the communication. EverySystem has an input — responsible for receiving messages from the applicationand forwarding them to the pESB, and an output — responsible for receivingmessages from the pESB and forwarding them to the application.

The Transform Routing Unit represents a special unit responsible for trans-formation and/or routing. Every TRU has an input and an output, which can beeither another TRU or a system. This approach gives great flexibility to the oneresponsible for configuring pESB to particular integration needs. A message fromthe output of one TRU might be sent to the input of another TRU. Moreover,this unit might be considered as an independent component — with input andoutput and business logic responsible for transformation and routing, what en-ables re-usage of that component in different integration projects or by multiplesystems.

Figure 7.5: pESB configuration model

7.7.2 Configuration example

To better understand the process of configuring pESB let us present an exampleof the pESB configuration. The example is based on the case study describedin the chapter ”Case Study” (6). Because the case study is complex and in-volves multiple systems, only an excerpt of the whole integration solution will bepresented in this section.

The presented integration solution will only involve the following systems:

1. Internet Shop — responsible for the interaction with the user and placing(and confirming) orders


2. Orders Fulfilment System (Order System) — responsible for fulfilling theorders

3. Storage System — responsible for providing information about productsupplies

One of the requirements of the proposed solution is that it must make anapplication independent of other applications’ message formats. A message for-mat change of one application cannot enforce the change of message formats ofapplications communicating with it. For example, an Internet Shop message for-mat change should not enforce a change in the Orders Fulfilment System messageformat.

Moreover, the business logic responsible for making a decision where particularmessage should be forwarded must be separated from the application. This shouldbe provided by the component, which might be changed independently of anyother application.

The above requirements can be fulfilled by the pESB using the followingconfiguration (Figure 7.6):

1. Systems — the input and output of the messages:

• InternetShop

• OrderSystem

• StorageSystem

2. Transform Routing Units — responsible for transformation of message for-mat:

• InternetShop

• OrderSystem

• StorageSystem

3. Transform Routing Unit named Router — responsible for routing of themessages basing on business rules

7.7.3 More on transformers and routers

A component configuration is designed in a plug-in architecture, which meansthat it is possible to extend a basic set of configuration methods with new ones.Transformer and router are actually names of the interfaces, thus in order tocreate new method of configuration — one must simply create a new class thatwill implement the interface. pESB comes with two implementation of the trans-former interface — the Java code and the XSLT file and one implementation ofthe router interface — the Java code. This feature gives the user the possibilityto decide using which technology particular component will be configured.


Figure 7.6: pESB configuration example

7.7.4 Performing the configuration

The process of configuring pESB consists of two steps:

• creating Transform Routing Units (with Routers and Transformers) andSystems

• creating connections between Systems and TRUs

The following snippet of code creates a Transform Routing Unit object withXSLT Transformer:

String xsltCode = "<?xml ...";TransformerDTO transformerDTO = new TransformerDTO(TransformerDTO.TYPE_XSLT);transformerDTO.setParameter(TransformerDTO.CODE, xsltCode);

TransformRoutingUnitDTO truDTO =new TransformRoutingUnitDTO("OrderSystem", transformerDTO, routerDTO);

Once Transform Routing Unit object has been created (variable truDTO) itmust be registered at pESB:

int result1 = bean.registerTransformRoutingUnit(truDTO);// (bean is EJB remote interface of pESB configuration bean)

Variable result contains the status of registration, 0 in case of a successand −1 in case of an error.

After creating the Transform Routing Units two System, systems must becreated:

SystemDTO systemDTO = new SystemDTO("OrderSystem");int result2 = bean.registerSystem(systemDTO);


The next step in the process of configuration is creating links between Trans-form Routing Units and Systems. The following snippet creates the appropriateconnections:

int result3 = bean.setTransformRoutingUnitInput(truDTO, systemDTO);int result4 = bean.setTransformRoutingUnitOutput(truDTO, systemDTO);

int result5 = bean.setSystemOutput(systemDTO, truDTO);int result6 = bean.setSystemInput(systemDTO, truDTO);

It is worth mentioning that the links should be created on both sides, which incontext of the above example means that the OrderSystem output and input mustbe connected with the TRU, and the TRU input and output must be connectedwith OrderSystem.

7.8 Integration design patterns supported by pESB

This implementation of the Enterprise Service Bus utilises the following integra-tion design patterns:

• Message (3.1) — data record used to exchange data between an applicationand pESB and also between internal components of pESB

• Message Channel (3.3) — in order to provide reliability (the guaranteethat no message will ever be lost) all the communication between internalcomponents of the pESB is performed using JMS Queues

• Message Router (3.4) and Message Translator (3.5) — this functionalityis provided by a user-defined Router and Transformer, which is a part ofa Transform Routing Unit (7.7.1). Because pESB does not put any restric-tions on the operations that might be performed on the received messageinside those components, a lot of variants of those two integration designpatterns might be created, such as the following:

– Fixed Router, Content-Based Router, Context-Based Router (3.4)

– Message Filter, Recipients List (4)

– Content Enricher, Content Filter (4)

Despite the simplicity of this integration solution it is also possible to createother, more complex, integration design patterns, such as the following:

• Normalizer (4) — this design pattern has been actually used to solve thecase study integration problem (7.7.2), the solution consists of multipleTransform Routing Units performing the role of Message Translators andone central Transform Routing Unit performing the role of a MessageRouter

• Message Broker (4) — pESB itself might be perceived as a Message Brokerpattern. pESB is a central component of the system where all message


routing operations are being performed. It also reduces the number of mes-sage channels, because an application in order to be able to communicatewith other systems needs only one message channel — to the pESB.

7.9 Problems encountered during the implementa-tion

During the implementation phase we realised that it is a very complex task toimplement functionality enabling receiving of the messages in the push mode —mainly because, such a method requires a distributed transaction. pESB afternotifying the client that there is a pending message would have to wait until theclient fetches that message. This would have to be done in a transaction, becausethe client currently may experience some problems and would not be able tofetch a message (or there might be a timeout on communication or some otherproblem). In such situations the message should be returned back to the queue.Having in mind the fact that pESB is an application running in an EJB container,where transactions are handled automatically by the transaction manager, thiscould be a challenging task. Moreover, the mechanism of notification wouldrequire pESB to connect to the client, In order to facility this the client wouldhave to have some port open, on which pESB could reach him. That in turnwould put additional requirement on the client, not only on the design of it,but also on configuration (i.e. it would not be possible to use the client behinda firewall). The solution to the above might seem to be initialising the connectionto the client from the pESB side, but that would complicate pESB and wouldrequire pESB to have a mechanism to reinitializing the connection after it hasbeen broken, because of the problems experienced by the client. Again, keepingin mind the fact that pESB is the application running in an EJB container thathinders the manual (performed by an application) management and sharing ofthe connections, because resource management is being done by the applicationserver, not directly by an application, it makes that solution very difficult toimplement.

Despite a lot of advantages of the push mode approach we have decided toimplement the pull mode.

Chapter 8

Summary

The integration problem has been well-known since the 70s. Since that timea lot of technologies and integration styles have been in use, some of them havebeen described in chapter 2. The latest approach to the integration problem isan Enterprise Service Bus, based on the concept of messages and asynchronouscommunication. Our software project — the implementation of an ESB — triesto fulfil the gap on the market of such products. The main focus in its develop-ment was put on the usage of well-known, well-tested, already existing productsand technologies available on the market, such as: Java, Java Enterprise Edition(Java EE) application server performing the role of a container (hosting platform)for an Enterprise Java Bean (EJB) application, Java Message Service (JMS) andXML — because, we believe that those technologies combined with the deliber-ated architecture will enable our product to be a reliable, efficient, secure and —the most important thing in the era of constantly growing IT systems — scalableintegration solution.

The domain of this paper — integration — usually concerns large IT systems,quite often systems, which are crucial for the operation of the company, thusintroducing a solution, which will not be of high quality is not an option. A highquality solution is the one which is reliable, secure, easily scalable, distributed andhaving high performance. Only such solutions might be created as an answer forthe integration problem. Because of the reasons mentioned at the beginning, thereis no place for not well tested, buggy, unreliable software products. This, as it iswell known in IT, is very difficult to achieve. To help in solving this challengingtask integration, the integration design patterns might come handy.

As it has already been mentioned in the introduction to this thesis, due to thetime and resources limitations our effort has been directed in such a way so thatthe created application would fit into an empty niche. This niche creates a spacefor a lightweight solution suitable to solve most common integration problems.For obvious reasons, it was not possible to create a fully functional software insuch a short time that would provide functionality comparable to the solutionsdelivered by large software vendors. Never the less, aiming for this niche madeit possible to create a fully functional software, and, at the same time, gaining

85

86 CHAPTER 8. SUMMARY

knowledge about the topic of application integration. The main assumptions thathave been made for the application have been outlined in the first chapter of thispaper. Now when it is finished and working, we can say that those assumptionshave been fulfilled. The ease of use and simplicity have been our main goals andthey have been achieved. We not only managed to describe the topic that wehave decided to undertake from the theoretical point of view, but we also usedthis theoretical knowledge while creating our application. We also managed toapply designed patterns presented in this paper in a practical way, in order toobtain the desired results.

Due to the shortage of time our application lacks a graphical user interfacethat would simplify its usage. Creating such a user interface would signifi-cantly speed up the process of designing an integration solution. This featuremight be one of the possible ways of the further development of this application.Another way is to enrich its functionality by adding a larger number of designpatterns for the user to choose from. While adding those new patterns, the over-all simplicity of the solution should be kept in mind, as it is one of the mostimportant features of this application, that cannot be lost during its further de-velopment.

The application that has been created during our work can be seen as a basesystem, which can be further extended in various ways — to improve its function-ality and ease of use. Sample ways in which it can be extended have been alreadypointed out in the previous paragraph. Those are of course only the extensionsthat in our opinion would contribute most significantly to the development ofour system. Other possibilities and ways of developing our application can alsobe applied. By gradually developing it and adding new features, a very powerfulintegration tool can be created, which can be used to solve not only the basicintegration problems but also more sophisticated ones. Of course, it will not beable to compete with solutions provided by the large IT companies, but over timeit can become an interesting alternative to complicated and expensive solutions.

Our work proved that the knowledge of integration design patterns is essential— not only for the designers and developers of integration solutions, but alsofor the people involved in creating IT products. Nowadays, even the simplestapplications, created to simplify everyday routines may one day be integratedinto a large computer software infrastructure. Awareness of the problems coveredin this thesis will enable to design and implement solutions — flexible enough toone day become a part of some other — larger — system.

Bibliography

[1] Christopher Alexander. A Pattern Language: Towns, Buildings, Construction.Oxford University Press, 1977. [cited at p. 43]

[2] Christoph Bussler. B2B Integration. Springer, 2003. [cited at p. 20]

[3] David Chappell. Enterprise Service Bus. O’Reilly Media, Inc., 2004. [cited at p. 53,

59, 60, 61, 62]

[4] Ralph Johnson John Vlissides Erich Gamma, Richard Helm. Design Patterns: El-ements of Reusable Object-Oriented Software. Addison-Wesley Professional, 1995.[cited at p. 43]

[5] Thomas Erl. Service-Oriented Architecture (SOA): Concepts, Technology, andDesign. Prentice Hall PTR, 2005. [cited at p. 21]

[6] Martin Fowler. Patterns of Enterprise Application Architecture. Addison-WesleyProfessional, 2002. [cited at p. 11]

[7] Bobby Woolf Gregor Hohpe. Enterprise Integration Patterns: Designing, Build-ing, and Deploying Messaging Solutions. Addison-Wesley Professional, 2003.[cited at p. 7, 17, 23, 24, 26, 27, 30, 32, 33, 35, 36, 38, 39, 44, 45, 46, 47, 48, 49, 50]

[8] William Grosso. Java RMI. O’Reilly Media, Inc., 2001. [cited at p. 11, 26]

[9] Stany Blanvalet Jeremy Bolie, Michael Cardella and Matjaz Juric. BPEL Cook-book: Best Practices for SOA-based integration and composite applications devel-opment. Packt Publishing, 2006. [cited at p. 52, 57]

[10] Ann Wollrath Sam Kendall Jim Waldo, Geoff Wyant. A Note on DistributedComputing. Sun Microsystems Laboratories, Inc., 1994. [cited at p. 27]

[11] Doug Kaye. Loosely Coupled: The Missing Pieces of Web Services. RDS Press,2003. [cited at p. 9]

[12] Susan Bishop Alan Hopkins Sven Milinski Chris Nott Rick Robinson JonathanAdams Paul Verschueren Martin Keen, Amit Acharya. Patterns: Implementingan SOA Using an Enterprise Service Bus. IBM Corp., 2004. [cited at p. 55]

[13] Ken Vollmer Mike Gilpin. The Forrester Wave: Enterprise Service Bus, Q4 2005.Forrester Research, Inc., 2005. [cited at p. 75]

87

88 BIBLIOGRAPHY

[14] Roy W. Schulte. Predicts 2003: Enterprise Service Buses Emerge. Gartner, Inc.,2002. [cited at p. 51]

[15] Kim Williams Scott McLean, James Naftel. Microsoft .NET Remoting. MicrosoftPress, 2002. [cited at p. 11]

[16] Venky Shankararaman Wing Lam. Enterprise Architecture and Integration: Meth-ods, Implementation and Technologies. IGI Global, 2007. [cited at p. 7]

Appendices

89

List of Figures

2.1 File Transfer style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.2 Shared Database style . . . . . . . . . . . . . . . . . . . . . . . . . . 242.3 Remote Procedure Invocation style . . . . . . . . . . . . . . . . . . . 262.4 Messaging style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.1 Design Pattern Message . . . . . . . . . . . . . . . . . . . . . . . . . 303.2 Design Pattern Channel . . . . . . . . . . . . . . . . . . . . . . . . . 323.3 Design Pattern Router . . . . . . . . . . . . . . . . . . . . . . . . . . 333.4 Design Pattern Translator . . . . . . . . . . . . . . . . . . . . . . . . 353.5 Design Pattern Endpoint . . . . . . . . . . . . . . . . . . . . . . . . . 383.6 Overview of a communication based on message design patterns . . . 39

4.1 Design Pattern Content-Based Router . . . . . . . . . . . . . . . . . 444.2 Design Pattern Message Filter . . . . . . . . . . . . . . . . . . . . . . 444.3 Design Pattern Dynamic Router . . . . . . . . . . . . . . . . . . . . . 454.4 Design Pattern Recipients List . . . . . . . . . . . . . . . . . . . . . . 454.5 Design Pattern Splitter . . . . . . . . . . . . . . . . . . . . . . . . . . 464.6 Design Pattern Aggregator . . . . . . . . . . . . . . . . . . . . . . . . 464.7 Design Pattern Routing Slip . . . . . . . . . . . . . . . . . . . . . . . 464.8 Design Pattern Process Manager . . . . . . . . . . . . . . . . . . . . 474.9 Design Pattern Message Broker . . . . . . . . . . . . . . . . . . . . . 484.10 Design Pattern Envelope Wrapper . . . . . . . . . . . . . . . . . . . . 484.11 Design Pattern Content Enricher . . . . . . . . . . . . . . . . . . . . 494.12 Design Pattern Content Filter . . . . . . . . . . . . . . . . . . . . . . 494.13 Design Pattern Normalizer . . . . . . . . . . . . . . . . . . . . . . . . 50

5.1 Overview of a communication based on RPC and MOM . . . . . . . 535.2 Tightly coupled system architecture compared to ESB . . . . . . . . 545.3 Conceptual overview of an Enterprise Service Bus components . . . . 595.4 ESB Design Pattern VETO . . . . . . . . . . . . . . . . . . . . . . . 595.5 Comparison of a typical transformer and Two-step XRef . . . . . . . 61

6.1 Case study systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676.2 Case study processing sequence . . . . . . . . . . . . . . . . . . . . . 686.3 Case study: Message Channel scenario . . . . . . . . . . . . . . . . . 70

91

92 LIST OF FIGURES

6.4 Case study: Message Channel with Router and Translator scenario . 716.5 Case study: Enterprise Service Bus scenario . . . . . . . . . . . . . . 72

7.1 Integration architecture layers . . . . . . . . . . . . . . . . . . . . . . 757.2 pESB architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 767.3 pESB communication architecture . . . . . . . . . . . . . . . . . . . . 777.4 pESB processing sequence . . . . . . . . . . . . . . . . . . . . . . . . 787.5 pESB configuration model . . . . . . . . . . . . . . . . . . . . . . . . 807.6 pESB configuration example . . . . . . . . . . . . . . . . . . . . . . . 82

List of Tables

2.1 Trade-offs between synchronous and asynchronous model . . . . . . . 22

3.1 Levels of data transformation . . . . . . . . . . . . . . . . . . . . . . 37

93

Index

Aggregator, 45asynchronous communication, 20

big-endian system, 13

command message, 30Content Enricher, 48Content Filter, 49Content-Based Router, 35, 44Context-Based Router, 35COTS, 7

data timeliness issue, 20Datatype Channels, 33Dead Letter Channel, 33Dead Message Queue, 33design pattern, 43document message, 30Dynamic Router, 44

Enterprise Service Bus, 51Envelope Wrapper, 48event message, 30

file transfer integration style, 22Fixed Router, 34Forward Cache Integration pattern, 62

integration style: Remote Procedure Invo-cation, 26

Invalid Message Channel, 33

legacy application, 7loose coupling, 9

message body, 30Message Broker, 47message channel, 32Message Endpoint, 38Message Filter, 44message header, 30Message Oriented Middleware, 52message properties, 30Message Router, 34

message sequence, 30Message Transformer, 36Message Translator, 36messaging integration style, 27

Normalizer, 49

Point-to-Point Channel, 33Process Manager, 47Publish-Subscribe Channel, 33

Recipients List, 45response-reply message, 30Routing Slip, 46

shared database integration style, 24single point of failure, 25small-endian system, 13Splitter, 45synchronous communication, 20

tight coupling, 10Transaction Management, 56Two-step XRef pattern, 61

VETO pattern, 59VETOR pattern, 60

95

Documents

Design patterns in application integration