JBoss Architect Meetup - December 2013 - JBoss Fuse in Vodafone’s Global Integration Gateway

Jboss Fuse in

Vodafone’s Global

Integration Gateway

5th December 2013 / Bruno Meseguer

Jboss Architect Meetup

Jboss Architect Meetup | 05 December 2013 2

Contents of the presentation

• Who we are within Vodafone

• The performance challenge

• Selection process for the new technology

• Creation of a new framework

• Some final thoughts

Welcome !

Jboss Architect Meetup | 05 December 2013

Global Integration Gateway (GIG) Vodafone Group Enablers


Vodafone Group Enablers

• Enablers team, responsible to enable capabilities in Vodafone Group

– Charging

– Messaging

– Customer Registry

GIG

– … and more

• GIG (Global Integration Gateway) (100+ systems)

– Intelligent Gateway which allows flexible Partners/Vodafone integration.

– Local/Global Partners and Vodafone internal systems can access services provided by OpCo’s.

– Opco’s can access services provided by Partners and Vodafone internal systems.

Who we are and what we do


Vodafone’s Global Integration Gateway (GIG)

HTTP HTTP

Technical capabilities view

Authentication Authorisation

Protocol Bridging

Orchestration Throttling

Routing

Header Enrichment

Error Handling

Data Transformation

Logging & Reporting

Asynchronous

Very fast > 800 TPS



HTTP HTTP

Bridge In

(north)

IBM DP

Bridge Out

(south)

IBM DP

Orchestration

(Tibco BE)

Atomic

(Tibco BE)

Atomic

(Tibco BE)

Orchestration

Orchestration

Orchestration

Atomic

Atomic

Atomic

System layers view


Vodafone Group Enablers

• In GIG’s ecosystem there are currently 100+ systems

• GIG’s current traffic is 1000+ TPS

• GIG’s original design aimed to be capable of handling 10,000 TPS

– This principle has been maintained on every evolutionary step of the solution

– GIG scales horizontally: 10,000 doesn’t represent a barrier

• The latency that GIG introduces is <0.1 seconds on slowest use cases

• The average payload size is around 2KB

– As GIG typically exposes API’s, payloads are not significantly big.

Some GIG numbers



• 2008:

– Poor HTTP performance from TIBCO component

– Had to find a fast HTTP gateway

– IBM's DataPower was the answer > 800 TPS

• 2012:

– Vision to port systems to a cloud environment

– IBM's hardware not portable

– the quest to find a replacement for IBM's DataPower starts

A little bit of history



• To find a software based solution, current DataPowers not portable to the cloud.

– With very high levels of performance, ideally > 800 TPS

– Fully asynchronous socket handling

– EMS compatible

• Minimise impact on clients.

– Able to maintain the same port definitions as in DataPower

• Minimise migration efforts

– Quick adoption of the technology

– 70+ flows to be migrated

The goals to replace IBM’s DataPowers


The performance

challenge


The performance challenge

• A normal HTTP server retrieves resources or generates some content and quickly responds to the

client

• What happens when a system acts as a pass-through and needs to wait for backend responses?

• delays from the backend could be disastrous to the system



• The worker thread-pool gets exhausted, where are the workers?

Quickly running out of workers!



• Worker threads need to wait for the backend response to resume.

Threads busy… doing nothing!



• A waiting queue of requests rapidly grows

Requests keep coming!



• Clients get increasingly angry!

Unhappiness!



• The system is exposed:

– Delays from the backend could be disastrous to the system

– The workers thread pool gets exhausted

– Worker threads need to wait for the backend response to resume

– A waiting queue of requests rapidly grows

• Systems will need to be proved to be robust:

– Under high rates of client requests

– Possible delays from back-ends

• Systems need to be Asynchronous


Selection Process


Selection Process

• Direct competitor to DataPower

• Highly recommended by Group Architecture

– Pre-selected by Group Architecture to replace DataPower

• Can be used as a Hardware or Software based solution

• A promise of raw performance, up to 30000 TPS

Layer7… first candidate


Selection Process

• How can we stress a system capable of 30000 TPS without the need of a farm of HTTP clients?

• There is an old friend to us called Netty used back in 2008 (a beta library back then) to test

DataPower on the lab.

• Using Netty we created a powerful client and stub proved to handle 5000+ TPS per instance

• Netty is an NIO client/server framework (event driven) with HTTP capability:

– Non-blocking socket management

– threads are freed up when posting requests, they don't wait for responses

– difficult to exhaust the thread pool

– responses are taken care of by a second thread pool

– the socket and HTTP context are kept in memory awaiting for a response thread to pick them up and close

Lab preparation


Selection Process

Lab preparation

• To stress the bridge we bombard it with a constant TPS rate and delay the response by 5 seconds.


Selection Process

• Within the space of two weeks we realised Layer7 was not the right choice

– By the third day of testing we aborted the performance study for Layer7

• Layer7 proved not to fulfil our performance requirements

– delays at the backend proved to be deadly to Layer7

– it was not asynchronous

Layer7 results


Selection Process

• As Netty proved to be very efficient we approached Red Hat and asked to provide us with Netty

support

• Initially Red Hat recommends Fuse + Jetty (Netty equivalent)

– 1st round of performance test show the solution works (400+ TPS), and it is indeed asynchronous

• From our Netty benchmarks we knew there was more potentially to exploit

• Red Hat agrees to integrate Netty HTTP into their product (it already included the Netty TCP

component)

– Red Hat delivers a Netty HTTP concept for us to try out

– 2nd round of performance show an improvement by a factor of 2 at least

Red Hat comes into play


Selection Process

• We triggered parallel assessments for:

– JBoss7

– TIBCO BusinessEvents 5.0

– WSO2

More candidates…


Selection Process

Performance results


Selection Process

Performance results


Selection Process

• Although only one product could get selected, all of them had an opportunity to identify

weaknesses in their product

• They all immediately put engineers at work to focus around those areas to be improved

• They all managed to boost up their engines:

– some more than others (x2, x3, x4)

– some adopted the asynchronous mechanisms (x100)

Everyone wins


Selection Process

• Fastest NIO product available (Netty enabled)

• Integration product (Camel)

• OSGI container

• Fully covering our migration requirements

• Java based solution

• Red Hat’s engineers: main open source committers

• Red Hat quality assurance

Red Hat’s JBOSS FUSE gets selected


Selection Process

• It was very critical to make to right choice

• We invested the right amount of time on each one of the steps

• We know we got it RIGHT!

No rushing !


Vodafone’s Framework



• Jboss Fuse offering an opportunity to improve DEV, Deployment & Support processes

– Traditionally our developments are based on frameworks

– Upgrades on frameworks require:

from DEV a new build (per component)

and from D&S another testing and deployment process

… per component !

Jboss Fuse’s potential



Traditional build and deployment processes

Customisation

Vodafone’s

Framework

Build Customisation

Vodafone’s

Framework

Deployable Unit

Deployable Unit

Source Code

Vodafone’s

Framework

Customisation Customisation

Customisation

JVM

JVM



• JBoss Fuse container is OSGI based

– It encourages to think differently

– Perhaps Vodafone can exploit the benefits

Could we change our processes to reduce efforts?

Could these benefits help the various involved teams

Development

Deployment

Support

OSGI and bundles



New processes: OSGI orientated

Build

Vodafone’s

Framework

Deployable Unit

Customisation

Deployable Unit

Source Code

Vodafone’s

Framework

Customisation Customisation

Customisation

Customisation

Deployable Unit

JVM

Customisation

Customisation

Vodafone’s

Framework



• The Framework is decoupled from business components and sits on the OSGI container as a

reusable module

• Implemented using Camel route definitions

• It contains 90% of the execution logic of a flow

• The Framework is responsible for

– Standarising flow definitions

– Reduce complexity in customisations

– Include all common functionality

– Ensure stability, robustness and performance

Framework's main bundle



Framework's main bundle

common common common

Direct-VM

JMS

common

Framework 1.0.0

1. Accept internal

invocations.

2. Throtteling.

3. Header

Enrichment.

4. System Logging.

5. Collect Reporting

Information.

6. Bridge to JMS.



• Business components (called Connectors) are configured to communicate with the framework

• Implemented using Camel route definitions

• Connectors are responsible for

– Listening to incoming requests

– Concentrate on the business logic

– Delegate execution control to the Framework

Connector’s bundle



Connector’s bundle

HTTP

custom

Direct-VM

out

Flow A

1. Open an HTTP

port.

2. Custom manipulation, typically

extraction of the Service ID, MSISDN,

and country code.

3. Invoke the Framework

route definition.



• Connectors are configured to communicate with the framework

• N connectors talk to the same single Framework module within the same OSGI container

• Upgrades on the framework do not require new deliveries from Development and do not require

redeployments

Framework and connectors interaction



Framework and connectors interaction



• Upgrades on the Framework do not cause an impact on Connectors

• When a Framework fix is delivered, Connectors are simple reconfigured

• Multiple Framework versions can coexist if necessary

Framework upgrades



Framework upgrades


Direct-VM

in

JMS

common

Framework 1.0.0

custom

HTTP

Direct-VM

out

HTTP

custom

Direct-VM

out


Direct-VM

in

JMS

common

Framework 2.0.0

Flow A

Flow B



• An early performance round shows a degradation (when

compared to POC results) well within the acceptable

margin

– around 10-15% loss only to the raw potential available

– with 5 seconds delay at the back and 2K payload we obtain 700

TPS

Note: The Lab environment used is SLOW !

We think the Lab environment is around half the speed of a

modern environment.

• An official data gathering is pending to be collected

Framework's performance

0

100

200

300

400

500

600

700

800

PoC Framework

TPS


Final overview


Current status of the project

• Vodafone’s Framework

– Implemented internally at

– Nearly 100% of the functionality completed

• Migration of DataPower flows

– The migration is outsourced to our trusted vendor Bulgaria

– Sprint 6 out of 8 is on its way

– 70+ flows in total

• Deployment

– First migrated flows started to be deployed in the pre-production environment


How JBoss Fuse compares to DataPower?

• Of course, JBoss Fuse is software based instead of Hardware based

– this means it is “cloudable”

– this means we are free to choose the hardware to run on

– it means the hardware is much cheaper (this is specially important when scaling)

– it means everyone can run it on its laptop

• Performance results suggest a substantial gain in overall TPS rate (1000+ TPS per instance)

• We can exploit the potential of 64 bits environments

– DataPower runs on 32 bits unless money is invested to acquire the new 64 DataPower XI52s

• The community of open source based solutions is much bigger


Are there any other advantages?

• it is Java based

– the skills required are much more common

• It is developer friendly

– DataPower development is painful

• One single Framework for all (instead of several in DataPower)

• RED HAT Customer support represents a quality leap coming from IBM's

– fast responses to customer cases

– skilled agents at the other end of the line

– proactive and collaborative reaction

– open to customer's point of view

– a user friendly Customer support Portal !!

Absolutely !!


What about the disadvantages?

• It's new

– we don't have OSGI experience

– we don't know yet the pitfalls

• New Component

– Red Hat has included Netty HTTP to satisfy our requirements

– we may find some integration defects

• New Framework

– will need some time to mature

– surely problems will be found

– migrated components may differ in behaviour from DataPower


The good and the bad

• The good

– Plenty has been said on previous slides

– The raw performance of the Camel framework and the Netty HTTP component

– The incredible number of available alternatives on how to do a particular task

– Very active product, with constant addition of features

– Customer support is first class

• The bad

– Not intuitive with the mix of available frameworks

– The available IDE to graphically define flows is not mature enough

– Having many options to do one task is seen as a positive, but in some cases is a negative

– Steep learning curve for less experienced developers


Questions and answers

Thank you

Jboss Architect Meetup | 5th December 2013

Technology

JBoss Architect Meetup - December 2013 - JBoss Fuse in Vodafone’s Global Integration Gateway