Lecture 3 Page 1 CS 239, Spring 2001 Interprocess Communications in Distributed Operating Systems CS 239 Distributed Operating Systems April 9, 2001

Lecture 3Page 1CS 239, Spring 2001

Interprocess Communications in Distributed Operating Systems

CS 239Distributed Operating Systems

April 9, 2001


Introduction

• Message passing communications

– Message passing paradigms

– Pipes and sockets

– Multicast and broadcast

• Request/reply communications

– RPC

– Secure RPC


Message Passing Communications

• By definition, distributed systems require IPC

• Most distributed systems connected only by network

• So most communications must be by message

– Other IPC abstractions built on messages


Levels of Abstraction of Communication

Interprocess Communication

Network OS

CommunicationsNetwork

Transactions

Request/Reply (RPC)

Message Passing

Transport Connection

Packet Switching


Basics of Messages

• Messages are collections of data objects– Header– Payload

• At message level, sender and receiver regarded as peers– Neither is in complete control


Steps in Sending a Message

• Sending process formats message• Invokes local OS to send it• Local OS (usually) copies it into kernel

space• Local OS transfers message to local

network interface• Network interface passes message to

network hardware


Steps in Receiving a Message

• Message delivered from network to receiver’s network interface

• Interface generates interrupt for OS• OS copies message to buffer • OS either interrupts receiver or waits for

receive system call• Message (usually) copied from OS buffer to

receiver process


Moving the Message

OS OS

Sender Receiver

2. Copy message into OS buffer3. Put the message in the network interface1. Create the message at the sender

send()

5. Transfer message to network interface6. Transfer to OS buffer

receive()

7. Transfer the message to receiver process4. Transfer message over network


Basic Message Primitives

• send(destination, message)

– Transfer contents of message to location specified by destination

• receive(source,message)

– Accept a message from source and put it in location specified by message


What is a Source/Destination?

• Complete specification of identity of a remote process

• Typically, a process name

– Often including a machine name

• Often with some specification of an internal address within the process


Common Source/Destination Naming Options

• Process name

• Link

• Mailbox

• Port


Process Name

• Varies depending on system specifics• Possible to have global, transparent

process namespace• More commonly, specify hosting node• Is this a good idea if processes move?• If only component of name, all

messages to a process share one queue


Symmetric Process Naming

A B

send(B,message1) receive(A,&message)

send(A,message)receive(B,&message)


Assymmetry in Naming

• Sender assumed to know receiver’s address

• Sometimes, receivers don’t know sender’s address ahead of time

– E.g., server handling multiple clients

• Sometimes receive() fills source into variable, rather than specifying value


Asymmetric Process Naming

A

B

C

send(C,message)

receive(&sender,&message)

send(C,message)

receive(&sender,&message)


Source/Destination Naming by Links

• Sometimes we want multiple communications paths between pairs of processes– Connections

• We can name these connections as sources and destinations– Links


Characteristics of Links

• Unidirectional• Multiple links allowed between two

processes• Each link independently nameable • Similar to multiple process entry points• Dynamically created and destroyed• Direct communications between peers


Links

A Blink1

link2link3

send(link1,message)create_link(B)create_link(B) create_link(A)

send(link2,message) send(link3,message)


Source/Destination Naming by Mailboxes

• Links aren’t appropriate for all cases• Sometimes sender doesn’t care about

identity of receiver– E.g., obtaining service from one of pool

of servers• Or receiver doesn’t care about identity of

sender• Mailboxes don’t require matching one

sender to one receiver


Mailbox Characteristics

• Multiple senders can put messages into the same mailbox

• Multiple receivers can retrieve messages from the same mailbox

• Varying semantics about how each of these works


Mailboxes

Mailbox

send(mailbox,message)

receive(mailbox,&message)

send(mailbox,message)

receive(mailbox,&message)


Message Synchronization and Buffering

• Sending and receiving a message requires some synchronization– When is it OK to send?– When is it OK to receive?– When is the process complete?– When can the system free various

buffers?


Blocking and Non-blocking Send and Receiver

• Blocking send blocks the sender

• Blocking receive blocks the receiver

• For how long?

– Some options here

• Non-blocking versions allow send and receive to go ahead immediately


Blocking at the Receiver

• In most systems, receiver blocks• Must wait for message to arrive before

proceeding• Sometimes, only blocks from moment of

message arrival until message is in receiver buffer

• Sometimes from point when receiver needs message


Blocking on Arrival

A B

Interrupt!Block!


Blocking on Demand

A Breceive(A,&message)

Block!


Common Options for Send Blocking

• Non-blocking send• Blocking send• Reliable blocking send• Explicit blocking send• Request and reply


Non-Blocking Send

A B

Block!


Blocking Send

A B

Block!


Reliable Blocking Send

A B

Block!

ACK

ACK


Explicit Blocking Send

A B

Block!

ACK

ACK


Request and Reply

A B

Block!


Buffering

• Any delay in duration of blocking implies a buffering capability– In the sending kernel– In the network– In the receiving kernel

• Buffering allows multiple simultaneous sends– Also provides reliability


Pipes and Sockets

• A common OS features providing message communications

• The interface between applications and the network

• Hides many details of the network


Pipes

• Originally built as IPC for single processor system

• Basically a FIFO byte stream buffer

• Bytes fed in one end

• Bytes read out the other end

• In the same order


Pipe Details

• Once set up, treated like a file• Typically, one process writes, another reads

– Usually parent and child processes• Send is non-blocking

– Assuming local buffer space available• Receive is blocking

– Receiver explicitly asks to receive bytes – And waits for them to arrive


Pipes, Buffering, and Blocking

• Pipes are implemented in the kernel by setting up a buffer– Of fixed size

• When writer fills the buffer, further writes block

• When reader asks for bytes from empty buffer, reads block


Named Pipes

• Normal pipes only usable by processes that share file descriptors– Usually parents and children

• Pipes can have names attached– In file system namespace

• Allows any process to name either end of pipe

• Implemented with kernel buffers or file system


Pipes and Distributed Processing

• Pipe implementations assume a shared buffer space

• Named pipes assume a shared namespace– And shared FS or buffer space

• Generally, distributed systems might not have these

• Can we provide a similar mechanism for that environment?


Sockets

• A general mechanism for distributed systems IPC

• Subsumes pipes

• Sets up communication channel named by two unique endpoints


Creating Sockets

• Both communicating processes must issue socket() calls

• Creating a logical communications endpoint (LCE)– Local to creating process

• Must bind LCE to a physical communications endpoint (PCE)


Example of Creating Sockets

A BsA = socket(domain, type, protocol)

domain describes protocol family to usetype describes high-level characteristics of communicationsprotocol specifies which protocol in the family to use

sB = socket(domain, type, protocol)


Binding Sockets

• A PCE is a network host address plus a transport port pair– Network host address space is global– Transport port pair numbers

generated locally• The bind() system call performs the

bindings


Example of Binding Sockets

A Bbind(sA,name,namelen) bind(sB,name,namelen)

Now A can send a message to B using the socket


Connected Sockets

• Bound sockets can be used for connectionless message sending

• Sending process must know remote PCE, though

• Connecting sockets allows send calls without specifying remote PCE

• Typically, for servers with well-known ports


Broadcast and Multicast

• Methods for sending the same message to more than one destination

• Broadcast typically sends to everyone on the network

• Multicast typically sends to chosen set of recipients


Best Efforts Multicast

• Guarantees delivery to at least one member of the group

• Useful when trying to obtain service from set of interchangeable servers


Reliable Multicast

• Guarantees delivery to all servers– Or none of them

• Useful for distributing shared information to cooperating processes

• If several reliable multicasts going on simultaneously, participants may see messages in different orders


Handling Failures in Multicast

• Sender expects some feedback from each recipient– Perhaps explicit acknowledgement

• If not all received, re-multicast– Or send to just those who didn’t

acknowledge• Failure of sender harder to deal with


Multicast Message Ordering

• FIFO: Messages from single source delivered in same order as sent

• Causal order: Causally related messages from different sources delivered in causal order

• Total order: All messages from all sources delivered in single order


Achieving FIFO Ordering

• Sender keeps local sequence counter

• Each multicast stamped with incremented counter

• Each receiver keeps counter for each sender

• Only deliver messages with expected timestamp

– Queue or reject others


Achieving Causal Order

• Use what amounts to vector clocks– One element for each sender

• Sender increments local element of vector each time it multicasts– Attaching new vector to message

• Each receiver stores vector of last message it accepted


Receiving Causally Ordered Messages Properly

• Compare incoming message’s vector to stored vector

• If one element of incoming vector is 1 greater than stored vector, accept it

• Delay it if more than one element greater, or any element greater by more than 1

• Reject if vector is less than stored one


Example of Receiving Causally Ordered Messages

0 0 0

0 0 0

0 0 01 0 0

1 0 0

1 0 0A

B

C

Should C accept this message?

Yes! 1 = 0+1, 0 = 0, and 0 = 0

But B’s message is lost

1 0 1

1 0 1

Should B accept this message?No! 1 = 0+1, 0 = 0, and 1 = 0+1

B has missed a causal message

1 0 0

1 0 0

1 0 0

1 0 1


Totally Ordering Multicasts

• Even harder than causal order• Essentially, requires a commitment protocol

– Everyone agrees they’ve received the message

– The sender knows everyone’s agreed– Then the message is delivered

• Related to 2-phase commit

Documents

Lecture 3 Page 1 CS 239, Spring 2001 Interprocess Communications in Distributed Operating Systems CS 239 Distributed Operating Systems April 9, 2001