Upload
johan-edstrom
View
859
Download
0
Tags:
Embed Size (px)
DESCRIPTION
From ChicagoCoders conference 2014
Citation preview
Johan EdstromSOA Executive and Apache developer.Apache Camel PMCApache ServiceMix committerOriginal CXF Blueprint AuthorCassandra client library [email protected]@apache.org
And debunking some SOA stuff
Using Apache Camel a bit like AKKA
Savoir Technologies - This is where we started
Where are we now?
Well, what happened there?
We kinda started focusing exclusively on scalability
• We went from the majority of our customers wanting “ESB”
• Or tune our “Messaging”
To
• Very modular software
• Infrastructure design that would last a while
• Key concept structures that were re-usable
• We’ve gotten quite a bit of experience here over the years
So what is that scalability we talk of?The right resources in the right place
• Tomcat doesn’t work better with 1024 threads
Messaging Writing QueueBrowsers to see where messages go isn’t awesome
State driven - Not mentioned in EIP Patterns (Yet)
Conversational - Not mentioned in EIP Patterns (Yet)
Effectively aggregating - Don’t use a RDBMS as a mutex
• select for update 2000 times a second doesn’t scale
Transactionally safe - And that doesn’t have to mean XA
Correctly utilizing EE
• Connectionpools aren’t for cowards
This is what we are trying to avoid....
What is the AKKA statement?
So where does AKKA come in?
It could be part of your deployment in Apache Karaf
It could be stand-alone
What is it that we really think is cool?
• Asynchronous systems!
• Actors
• Supervisors
• Scaling
• Immutability
Is that the goal for every Java shop out there?
So what does that AKKA look like?
Or maybe a bit more complicated?
So what is going on here?
• ActorSystem - umbrella naming
• System bound error handling
• Logging
• Scaling
• Then we introduce some EIP Patterns like “stuff”
We RoundRobin and allow it to scale
And we finally mount it
Are you trying to write Camel stuff?
Sure
• And yes I know that Camel-AKKA exists or is it AKKA-Camel now?
• I know there is a Scala DSL for Camel - fixed a few things there…
• What about the Scalaz stuff?
So…. That was really cool. But….
All projects aren’t greenfield
• Can we refactor?
We have existing staff and investments
• Is it easy to train
We want to do this in pieces
• We really want to try, test, promote
Could we do this with the existing EE Infra we have?
Let’s pretend you have Apache Karaf
This is not far from an EE container
• Add OpenEJB if you want and you’ll have a full stack
It contains a web-services layer
Dependency injection services
OSGi integration and legacy (JBI) support (Don’t use that. - Seriously…)
Camel for routing, transformation and mediation
Aries and Spring for EE support, JNDI, TX, JPA
Web-app deployment support
And we’ll add Apache ActiveMQ
This is another presentation :)
• Jeff did that one earlier today, I hope you did attend!
And I presume ya’ll know Apache Camel
Camel is a Domain Specific Language (DSL) focused on
implementing Enterprise Integration Patterns (EIPs)
• Examples: multicast, splitter, content-based router, routing slip, “dynamic
routers”, aggregator
EIPs are implemented by Camel “Processors”
• Users can develop their own processors
• A processor is a fundamental building block
• Bean language and bindings exists so that not a single piece of Apache Camel
Imports will be necessary when integrating your existing code
Camel can connect to a wide variety of integration technologies
• Examples: JMS, HTTP, FTP, SOAP, File - There are ~ 180 components
• Integrations are implemented by Camel “Components”
• Users can develop their own components
Apache Camel
Do I need all of that?
Nope, many solutions will need just a few things
• jaxb, camel, jms, soap, rest and perhaps jdbc
• Cut the container down to fit your needs
• We don’t need to load all of the 100+ Apache Camel components
• Pick and choose!
Should I run that messaging solution inside the “ESB”
• Entirely up to you, let us look a little deeper at that in a sec.
Can I test these solutions or am I stuck with
System.out.println and a remote debugger?
I’ve heard that OSGi sucks?
Yes, It is a bit more complex to manage a full lifecycle
• On the other hand you get reliable hot-deployment and things like
configuration, service management and service subscriptions out of the box.
Yes, package level import export can be confusing
• So make sure you use packages in a nice modular way.
Yes, it takes a while to learn the tooling
• Then did you wake up one day just knowing that WAR or EAR layout?
Yes, you need to learn the classpath and a bit about
classloading
• Which’ll make you a better developer in the end of the day.
• Many problems in this area actually are created by applying EE classloaders
to an OSGi environment.
Decoupling with OSGi
Enforces knowledge of imports and exports
Allows us to version
Programming model with micro-services allows
for re-use on an Api level vs code level
Promotes contracts
Use case #1
We want to accept incoming SOAP requests
We need to inspect those HTTP headers and then use them
to route
We want this to be dynamic and flexible We might want to add a more complex routing engine later
We want this to be modular and possible to run across
multiple systems
Verify your use-cases!
UseCase #1 outline
Let us use some EIP pattern symbols to outline this
• We haven’t really made any technology statements here
We are going to use Apache ActiveMQ because I said so.
It could easily have been any other JMS provider.
• We know we’ll have a synchronous part
Web-services by nature are.
• We’ll try and make all the other parts asynchronous
• We’ll also make sure that the participants don’t need strong integration or
visibility of each other, thus we’ll (hopefully) end up with some pretty re-
usable modules
• Another good approach is to also use sequence diagrams
What else do we want here?
A “Dynamic” Router!
• We’ll craft a simple header based Dynamic router from eip patterns.
Technically it is in between a routing slip with termination and a real
control channel pattern, we utilize JMS as our control channel for this.
We use this example to play with destinations from SoapUI
Remember that ActorSystem in AKKA?
• This dynamic router is a big part of “pretending” that we are doing that
The “Dynamic Router”
Yep, it is just a Java class.
• Imagine how easily you could integrate this with.... Anything!
We are able to turn “endpoints”, “routing”, “systems” or
whatever you want to call it into simple destination names.
We only need one route…..
We can break all these modules apart
Why build that type of a route?
So let’s sum up what we have at this point
We defined a CXF (JAX-WS) Endpoint, generated stubs and
tied it into Camel
• The MEP (Message Exchange Pattern) is IN/OUT since we use CXF
We then dropped in a router “Our Dynamic Router”
• It’ll listen to a JMS queue called recognizer
• Then it’ll start inspecting headers for a location
What about that testing?We are going to be dealing with an OSGi environment.
• Like an EE container it makes testing a tad less fun....
Full blown container testing with something like SoapUI
• Can be scripted and automated in Maven
• Or you could BDD / Cucumber script it
Full OSGi tests with Pax-Exam
• Allows you to wire up a full container
Simple OSGi registry tests with PojoSr
• Let’s you simulate an environment so you can test BluePrint!
Functional Junit/TestNG tests with Camel-Test.
• Should be combined with at least one of the above!
Testing the router
First of, let us just do a Camel Test, we want to verify that
the Dynamic routing works as expected.
• As you can see, none of these routes know about each other.
And if you can’t see that, they still don't.
Keep this in mind!!
How we’ll be using this dynamic data
We basically can use this to write completely dynamic steps
that will restart on a configuration change from
• File changes
• Code changes
• JMX changes
Proof of concept outline
We now have the following modules
• customer-service : A model library
• customer-service-ws : The webservices endpoint
• dynamic-routing : A generic module for routing
• response-builder : A simple module that returns a payload
• That means that all of the needed patterns are fulfilled.
• Competing consumer is a natural thing in JMS - KACHIIIIING!!!!
• Eventing is a natural part of Apache Camel - KACHIIIIING!!!!
• Request Reply is “Handled for us” - KACHIIIIING!!!!
• Dynamic config - KACHIIIIING!!!!
So why didn’t we just slap a camel route in Tomcat?
That’s why.
Otay - so what the heck did that have to do with AKKA
I’m glad you asked!
• We have, thanks to putting it on ActiveMQ
Immutability
Scalability
Asynchronous behavior
• We have a separation of Concerns - Kinda like the Actor
Are we really there tough?
• Not yet!
Do we think we can make that “Actor”
Lets make it very generic
Lets make it really dynamic
Lets use the power of OSGi to “do stuff”
The basic premise now is -
• JMS is our “Actor System”
• Camel handles Messaging and in/out for us as well as a lot of nice to have
stuff like errorhandling, graceful shutdown, startup, validation and so on.
• We want a “generic” way of handling say… Objects?
Second use-case!!!We are gonna take what we learned and build parts of a
dynamic ETL system - Basically CSV files
What is a mutator in this ETL system?
And how is it working?
Yey! We are manipulating strings!
• OMG, Such data….
• Kidding aside - we now have a tool where all the transformations can be
described, this system contains OGNL, pattern matchers, regular expression
engines and as we can see the MutationConfig contains factories for
invoking these guys.
Lets stick these in Camel (MutationManager)
See that Supervisor destination?
The “brian” of the whole system.
• Checks if this is an initial record
• Checks if there is pre-cleansing needed
• Then goes onto in some sub-processors talking to rules and persistence to
build a chain of MutationSets
Then we add an OSGi - Manager (ServiceFactory)
This is purely config driven in OSGi
And the Master system initiation
And showing all the moving parts
Running the system
How did it get to Mock?
System wide “Role” for that particular incoming source file
was established, it was added last in line.
The roles are just written to a Cassandra layer, the
supervisor and the MutationManager executes them.
And the operations?
Runtime loaded
• Part of a config chain - Applied as roles in a processing chain
• Processing chain determined from config and built by the ServiceManager(s)
• Can be hot deployed and loaded
• Can be changed on running processing
End result?
100% Event driven
100% Asynchronous
No camel knowledge really needed
Competing consumer - Horizontally scalable
Hot-Deployable
• And most of the cool stuff I can’t even show you…..
• This pattern is quite well documented in OSGi, not that often combined with
Camel but you end up with all of the parts that really are interesting in
something like AKKA
And what else is Savoir doing?https://github.com/savoirtech/aetos
• Cooler than Karaf :)
https://github.com/savoirtech/hecate
• Hector based Pojo Mapper
Object Graph
Annotations
• CQL3 based Pojo Mapper
Object Graph
Annotations
True collection handling
column adapters
https://github.com/seijoed/hazelcast-discovery
• HazelCast based ActiveMq discovery system
We might be releasing
Polymorphic CRUD
• Based on CXF
• Fully Cassandra (or whatever) backed
• Event driven
• Single bean wiring with entity Class
And we might be looking more at AKKA :)
Questions?