Azure Service BusCloud-scale Messaging and Events

Azure-in-the-ATL

December 2014

• Third Tuesday of the month, generally 6-8 PM

• Microsoft offices in Alpharetta

• Updates on www.meetup.com/Azure-in-the-ATL

• Formats• 1+ hour, single speaker

• 2 or 3 “lightning” talks (20-30 mins each)

• Panel discussions

• ?

About our group

Be a sponsor!

Agenda

• Messaging and Events – What and Why?

• Azure Service Bus overview

• Queues

• Topics

• Relay

• Event Hubs

A Brief History of Distributed Software Systems• In the beginning, it was all punchcards

• And then we got…• Operating systems

• Multiple CPUs

• Multitasking

• Threads

• Server farms and data centers and clouds and stuff

• Stupid cat pictures• browsers and web servers and stuff

Stuff Over Here Wants to Talk to Stuff Over There• Which means we got…

• Ethernet• RPC (DCOM, CORBA, etc.)• Enterprise Messaging (TIBCO, etc.)• XML messaging• SOAP• REST

• Most of these fall down at scale due to conceptual and/or practical limitations

• Messaging and Events provide a useful set of core concepts to reason about moving data around at scale (“in the cloud”)

Messaging and Events

• Messaging – transmission of data from sender to receiver via some intermediary• Implications

• Sender and receiver don’t necessarily know about one another

• Intermediary can be arbitrarily complex and interesting (multiplexing, durability, security, etc. etc.)

• Data goes only one way

• Events – data transmitted from sender to receiver as a side effect of some interesting occurrence

Azure Service Bus – Big Picture

• Umbrella technology that implements (most) messaging for Azure• Scales to potentially millions of concurrent message senders (Event Hubs)• Supports various standard messaging patterns

• Producer/consumer (competing or cooperative)• One-way async• RPC-ish• Reactive-style streamed event consumption

• Scenarios• IoT• Consumer messaging• MSMQ or TIBCO type messaging• Hybrid cloud

Azure Service Bus – Queues

• FIFO brokered messaging technology• Similar to MSMQ in usage (transactions, security, sequencing, etc.)

• Scenarios• Rate leveling

• Independent scaling of application tiers

• Fault tolerance and enhanced reliability

• Asynchronous processing

• Differences with Azure Queues• http://msdn.microsoft.com/en-us/library/azure/hh767287.aspx

Azure Service Bus – Topics

• Brokered pub/sub messaging• One-to-many fanout scenarios

• Topics are where messages are sent

• One or more subscriptions are created against a specific topic• Can be all messages, or filtered

• Scenarios• Domain Event pattern

• Scenarios where multiple consumers should process the same message

Azure Service Bus – Relay

• Niche technology that enables secure communication between private and public clouds

• WCF-based• Uses custom WCF bindings and behaviors

• Integrates with Azure AD Access Control

• Can be very useful for securely exposing legacy WCF endpoints to cloud-hosted code

Azure Service Bus – Event Hubs

• Implements very high-scale message ingest and consumption• Supports AMQP and HTTP APIs

• Also various language bindings (.NET, Java, etc.)

• Event delivery to readers is async via Consumer Groups (logical stream copy, similar to topic subscriptions)

• Uses partitions (within a CG) to logically segment data for fast consumption• Only 1 reader per partition• # partitions == # of readers in a group

• Readers can use per-partition checkpoints to mark progress and provide resiliency

• EventProcessorHost class simplifies administration of reader tasks

Demo – Service Bus Topics and Event Hubs

Not TicketMaster

Service Bus Event Hub

InventoryManagement

Auditing Analytics

Service Bus Topic