Upload
bilgin-ibryam
View
782
Download
1
Embed Size (px)
Citation preview
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears
June 2016
Bilgin Ibryam
@bibryam
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears2
Bilgin Ibryam
● Senior Middleware Architect at Red Hat UK
● Apache Camel Committer and PMC member
● Apache OFBiz Committer and PMC member
● Author of Camel Design Patterns (new)
● Author of Apache Camel Message Routing
● Twitter: @bibryam
● Email: [email protected]
● Blog: http://ofbizian.com
● LinkedIn: http://www.linkedin.com/in/bibryam
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears3
Apache Camel Project StatusIt has all necessary ingredients for a successful open source project.
● Community: 52 committers, 903 users
● Support by large vendors (Red Hat)
● Connectors (256), DataFormats (40)
● Enterprise Integration Patterns++
● Domain Specific Language
● Ecosystem: Karaf, ActiveMQ, CXF, Fabric,Hawtio, Spring and others
● Monolith, SOA, Microservices, Serverless
Source https://www.openhub.net/p/camel/
Stats Date: 06/06/2016
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears4
Application Integration with CamelWhat do you need to know to create great Camel applications?
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears5
Happy Path ScenariosHow Pipes and Filters Pattern looks like in Camel?
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears6
VETROWhat is a typical processing flow for a Camel route?
● Validate: validation, schematron, MSV, Jing, bean validation components
● Enrich: enrich and pollEnrich EIPs, custom beans
● Transform: Data formats, auto type conversion, templating components
● Route: Message routing EIPs
● Operate: this is the essence of the processing flow
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears7
Edge ComponentLet's start with a simple Camel route that consumes files
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears8
Edge ComponentHow to expose the same business functionality to multiple consumers?
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears9
Edge ComponentEncapsulate endpoint-specific details and prevent them from leaking intothe business logic of an integration flow.
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears10
Read vs Write OperationsHow to evolve Read and Write operations independently?
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears11
CQRSThis decouples read from write operations to allow them to evolveindependently.
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears12
Unhappy Path ScenariosHappy paths are the easy ones. More work is required for designing andimplementing the unhappy paths.
● Data Integrity Pattern ● Saga Pattern● Retry Pattern● Idempotent Filter Pattern ● Circuit Breaker Pattern● Error Channels Pattern● Throttling Pattern
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears13
Data IntegrityHow hard can it be to copy files from one location to another?
Download Data Integrity Chapter: http://bit.ly/came-design-patterns-sample
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears14
Data IntegrityTransactional systems
Local transaction manager
Global transaction manager
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears15
SagaHow to avoid distributed transactions and ensure data consistency?
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears16
SagaEnsures that each step of the business process has a compensating actionto undo the work completed in the case of partial failures.
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears17
RetryTo enable applications handle anticipated transient failures by transparentlyretrying a failed operation with expectation it to be successful.
● Which failures to retry? ● How often to retry? ● Idempotency ● Monitoring ● Timeouts and SLAs
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears18
RetryCamel RedeliveryPolicy
● The most well known retry mechanism in Camel● Retries only the failing endpoint ● Fully in-memory● Thread blocking behavior by default● Can be asynchronous● Good for small number of quick retries (in milliseconds)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears19
RetryActiveMQ consumer RedeliveryPolicy
● Retries the message from the beginning of the Camel route● Not used very often, but enabled by default● Fully in-memory● Thread blocking by default● Good for small number of quick retries (in milliseconds)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears20
RetryActiveMQ Broker Redelivery
● ActiveMQ specific and requires custom logic● It will consume the message again from a queue● Persisted at the broker rather than application memory● Can be clustered and use fail over, load balancing, etc● Good for long persisted retries (in minutes or hours)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears21
Circuit BreakerHow to guard a system by cascading failures and slow responses fromother systems?
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears22
Circuit BreakerImproves the stability and the resilience of a system by guarding integrationpoints from cascading failures and slow responses.
Closed state
Open state
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears23
Circuit BreakerImproves the stability and the resilience of a system by guarding integrationpoints from cascading failures and slow responses.
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears24
Circuit BreakerTwo Circuit Breaker Implementations in Camel 2.18
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears25
BulkheadHow to enforce resource partitioning and damage containment in order topreserve partial functionality in the case of a failure?
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears26
BulkheadEnforces resource partitioning and damage containment in order topreserve partial functionality in the case of a failure.
● Multi-threaded EIPs: Delayer, Multicast, Recipient List, Splitter, Threads,Throttler, Wire Tap, Polling Consumer, ProducerTemplate, and OnCompletion.
● Async Error Handler ● Circuit Breaker EIP
Possible Camel bulkhead points:
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears27
Scalability ScenariosVertical scaling (performance tuning)
● Endpoints: messaging client buffers, DB client batching, template caching choices● Concurrent consumers option: Seda, VM, JMS, RabbitMQ, Disruptor, AWS-SQS● Data types choice: affects content based router, splitter, filter, aggregator● Multithreading: parallel processing EIPs, threads DSL construct, Seda component,
asynchronous redelivery/retry● Micro optimizations: log tuning, camel sampler EIP, disable JMX, disable message
history, disable original message record● Startup/Shutdown: Use lazyLoadTypeConverters for a faster application startup, or
configure the shutdownStrategy for a faster shutdown● Tune: JVM options, networking and operating system
Camel performance tuning blog post: http://bit.ly/camel-tuningCamel performance tuning blog post: http://bit.ly/camel-tuning
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears28
Horizontal ScalingService Instance Pattern for accommodating increasing workloads.
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears29
Service InstanceAreas to consider before horizontally scaling a Camel application.
● Service state: load balancer, circuit breaker, resequencer, sampler,throttler, idempotent consumer and aggregator are stateful EIPs!
● Request dispatcher: Messaging, HTTP, file (what about locking?)
● Message ordering: exclusive consumer, message groups, consumerpriority, message priority, virtual topics
● Singleton service requirements: for batch jobs, and concurrent polling
● Other resource contention and coupling considerations
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears30
What did we cover so far?
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears31
How patterns are changing?What is happening in the IT industry today?
● Canonical Data Model● Edge Component● Reusable Route● Runtime Reconfiguration● Singleton Service● Batch jobs in JVM
● Bounded Context● Standalone services● Favor code duplication● Less configuration, more redeployment● Container managed singleton● Container scheduling ● Circuit Breaker, Bulkhead, Health checks..
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears32
Win a print copy of Camel Design Patterns
When was the first commit to Apache Camel project done?
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears33
Win a print copy of Camel Design Patterns
When was the first commit to Apache Camel project done?
More Information
Learn more about Apache Camel: http://camel.apache.org
Check out Camel Design Patterns: https://leanpub.com/camel-design-patterns
Develop Apache Camel based integrations using Red Hat JBoss Fuse: red.ht/FuseDev