Distributed RPC in Nova with ZeroMQ

04-16-2012Distributed RPC with ZeroMQ: code & analysis

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionCCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution

Distributed RPC in Novawith ZeroMQ

X

Image: http://ken.pepple.info/openstack/2011/04/22/openstack-nova-architecture/

1

Tuesday, April 17, 12

http://ken.pepple.info/openstack/2011/04/22/openstack-nova-architecture/

http://ken.pepple.info/openstack/2011/04/22/openstack-nova-architecture/



Etherpad

2

http://etherpad.openstack.org/FolsomDistributedRPC








@ewindisch

3


CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution

Etherpad









Brokered Messaging

Message QueueBroker

Application

5




Brokered Messaging

Message QueueBroker

Application

X

5



Brokered Messaging

Message Queue

Application



Brokered Messaging

Application



High-Availability

HA Failover

Application



failover fails

Application



DecentralizedApplication

APINova Scheduler

Nova Network

Nova Compute

Message Queue

Message Queue

APINova Scheduler

Nova Network

Nova Compute








APINova Scheduler

Nova Network

Nova Compute

Message Queue

Message Queue

APINova Scheduler

Nova Network

Nova Compute



Decentralized

AbandonedMessages

Application

APINova Scheduler

Nova Network

Nova Compute

Message Queue

APINova Scheduler

Nova Network

Nova Compute




APINova Scheduler

Nova Network

Nova Compute

Message Queue

APINova Scheduler

Nova Network

Nova Compute



Solution:

Distributed RPCfor OpenStack Nova



Peer2Peer

Application

API

Nova Network

Nova Compute

Nova Network

Nova API

Nova Compute

Nova Scheduler



Peer2Peer

Application

API

Nova Network

Nova Compute

Nova Network

Nova API

Nova Compute



Peer2Peer

Application

API

Nova Network

Nova Compute

Nova Network

Nova API

Nova Compute

Nova Scheduler

Nova Scheduler



Peer2Peer

Application

API

Nova Network

Nova Compute

Nova Network

Nova API

Nova Compute

Nova Scheduler



rpc.cast(topic.host)

Invoker(e.g. api)

PublisherDirect Topic

Consumer

PUSH -> PULL

Worker(e.g. compute)

Cast to host.




Invoker(e.g. api)


Consumer

PUSH -> PULL


Cast to host.




Invoker(e.g. api)


Consumer

PUSH -> PULL


Reply.




Invoker(e.g. api)


Consumer

PUSH -> PULL


Reply.


def _send(...):

“”” code omitted “””

# Curry the original request into a reply method.

orig_payload = [RpcContext.marshal(context), msg]

payload = [RpcContext.marshal(context), {

'method': '-reply', # Send original call to wrapper

'args': {

'msg_id': msg_id,

'context': RpcContext.marshal(context),

'topic': reply_topic, # our return path

'msg': orig_payload

}

}]


# Messages arriving async.

msg_waiter = QueueSocket(

"ipc:///var/run/nova/zmq_reply_queue",

zmq.SUB, subscribe=msg_id, bind=False)

try:

with Timeout(timeout):

with Timeout(30) as t1:

conn.cast(msg_id, topic, payload)

# Blocks until receives reply

responses = pickle.loads(msg_waiter.recv()[-1])


def reply(...): # this is -reply

"""Reply to a casted call"""

child_ctx = RpcContext.unmarshal(orig_payload[0])

response = ConsumerBase.normalize_reply(

self.get_response( # execute original call

child_ctx, proxy, topic, orig_payload[1]),

ctx.replies

)

_multi_send("cast", ctx, reply_topic, {

'method': '-process_reply',

'args': {

'msg_id': msg_id,

'response': response

}

})


def process_reply(...):

"""Process a reply"""

# zmq_replies is the receiver service

self.msg_waiter.cast(str(msg_id), str('zmq_replies'), response)



rpc.call(topic.worker)

Invoker(e.g. api)


Consumer

PUSH->PULLcurries reply method

PULL<-PUSHrpc.cast -process_reply() Worker

(e.g. compute.host)

Call method.




Invoker(e.g. api)


Consumer



(e.g. compute.host)

Call method.




Invoker(e.g. api)


Consumer



(e.g. compute.host)

Call method.




Invoker(e.g. api)


Consumer



(e.g. compute.host)

Call method.



“scheduler.host1” goes to “host1”

Where do messages for “scheduler” go?

Bare Topics



MatchMaking



RewriteEngine• Every incoming topic is

mapped to an array of topics and hosts.

• Usually returns a single topic/host.



Search for hosts

• Can be provided by:

• hash table (implemented)

• name service (i.e. DNS)

• directory service (i.e. LDAP)

• zookeeper

• etc...



rpc.fanout_cast(topic)e.g. "compute"

Invoker(e.g. api)


ConsumerPUSH -> PULLi.e. rpc.cast(topic.worker)

Worker(e.g. nova-compute on

compute.host)

MatchMaker

get_workersas workers

for each worker:

Fanout may have many workers.

Array *topic.worker

Fanout Cast.Tuesday, April 17, 12



Invoker(e.g. api)




compute.host)

MatchMaker


for each worker:


Array *topic.worker




Invoker(e.g. api)




compute.host)

MatchMaker


for each worker:


Array *topic.worker




Invoker(e.g. api)




compute.host)

MatchMaker


for each worker:


Array *topic.worker




Invoker(e.g. api)




compute.host)

MatchMaker


for each worker:


Array *topic.worker




Invoker(e.g. api)




compute.host)

MatchMaker


for each worker:


Array *topic.worker




Invoker(e.g. api)




compute.host)

MatchMaker


for each worker:


Array *topic.worker




Invoker(e.g. api)




compute.host)

MatchMaker


for each worker:


Array *topic.worker




Invoker(e.g. api)




compute.host)

MatchMaker


for each worker:


Array *topic.worker




Invoker(e.g. api)




compute.host)

MatchMaker


for each worker:


Array *topic.worker




Invoker(e.g. api)




compute.host)

MatchMaker


for each worker:


Array *topic.worker



http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.htmlhttp://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html


http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html









NOVARPC





Thank You

Slide deck:

http://bit.ly/HJogQ8

Essex Plug-in Download

https://github.com/cloudscaling/nova-mq






Technology

Distributed RPC in Nova with ZeroMQ