LECTURE 7: WEB SERVICES EVENT BASED SYSTEM …rbrennan/CA4006...• Event-Sourcing – Event store as the principle source of truth e.g. git version control – Provides strong audit

LECTURE 7: WEB SERVICES EVENT BASED SYSTEM PATTERNS – PART 3

CA4006 Lecture Notes (Rob Brennan and Martin Crane 2019)

1

Patterns Revisited

2

Event Driven Architecture Patterns

• “Architecture pattern promoting the production, detection, consumption of, and reaction to events” - Wikipedia

• Event = a significant change of state e.g. something has happened in the outsideworld due a user, sensor or other application– Events occur at a point in time– Events can have metadata or context e.g provenance

• Notification = the message that carries the data about the event (often confused with the event itself)

• Applications: user interfaces, IoT, I/O, distributed systems, realtime systems• Creates loosely coupled software components and services:

– Sender does not need to know the receiver (e.g. Publish-Subscribe)– Execution is decoupled from invocation– Can have multiple receivers for the same event– Receiver can re-order, prioritize, etc– Enables reconstruction of system state (if all interactions are via events) at specific

points in time


3

Basic Components


4

Event Dispatcher

Event Handler Function

(Subscriber)


(Subscriber)


(Subscriber)

Event Generator Function

(Publisher)


(Publisher)


(Publisher)

Martin Fowler’s Event-Based Patterns

• Actually event-based systems encompass many system patterns that can be broken down further*

• Event Notification– Communicating the existence of state changes– No response to sender– Risk: large-scale coupled flows that you are unaware of, hard to debug

• Event-Carried State Transfer– Sharing state, e.g. REST, for reduced latency and increased resilience– Risk: High message/storage overhead for the copied state, greater complexity for

receiver as they have to determine aggregate state

• Event-Sourcing– Event store as the principle source of truth e.g. git version control– Provides strong audit capability– Risks: re-playing events may depend on external systems, handle event schema changes

• CQRS (Command Query Responsibility Segregation)– Separate processing or data structures for Reads and Writes– Risks: more complex


5* https://martinfowler.com/articles/201701-event-driven.html

Handling Events: Traditional Programming Techniques

• Finite State Machines– But reality often leads

to state explosion

• Can specify behaviourin terms of messagesand state transitions SDL


6

FSM for TCP connection release From: http://www.opentextbooks.org.hk/ditatopic/3577

Example Node.js Event Loop


7

Programming Drawbacks

• Event-driven model adds indirection– Harder to find all the event handling code for a specific event– More complex model to hold in your head– Important to avoid multiple dispatchers

• Harder to re-factor events– Change event format => all receivers have to change (what if they are on

different machines/code-bases?)

• Performance can suffer, to process a message the dispatcher must– Find the right hander (lookup data struct)– Build message pipeline for handler (memory alloc)– Dynamically call handlers (more indirection possible)=> Slower than a function call


8

References

• Wikipedia, Event Driven Architecture, https://en.wikipedia.org/wiki/Event-driven_architecture

• M Fowler, What do you mean by “Event-Driven”? https://martinfowler.com/articles/201701-event-driven.html


9

LECTURE 8: SAFE ACCESS TO DISTRIBUTED SHARED RESOURCES: TIME, SYNCHRONIZATION, REPLICATION & CONSISTENCY

10CA4006 Lecture Notes (Martin Crane 2018)

Lecture Contents• Introduction

• Time in Distributed Systems:– Physical Clocks; Logical Clocks: Totally Ordered Multicast

– Lamport’s Algorithm: Vector Clocks: Causally Ordered Multicast

• Mutual Exclusion in Distributed Systems:– Centralized & Decentralized Solutions

– Election Algorithms

• Consistency Algorithms:– Sequential & Continuous Consistency

– Causal Consistency

– Client-Centric Consistency

• Replication & Caching– Client- & Server-initiated caching

CA4006 Lecture Notes (Martin Crane 2018) 11Lecture 8: Safe Access to Dist’d Shared Resources

Introduction• Distributed systems have unique challenges, e.g. synchronizing data &

resolving conflicts.

• Must replicate content but such replicas must be kept consistent with each other.

• Saw above how processes communicate – related to this is how they cooperate & synchronize with each other.

• Here, mainly look at how processes can synchronize:– So, vital that multiple procs don’t simultaneously access shared resource, but

cooperate to grant each other temporary exclusive access.

– Multiple processes may also need to agree on event orderings, e.g. message from process 𝑃 sent before/ after another from process 𝑄

• Synchronization in DS thus much harder than synchronization in uniprocessor or multiprocessor systems.

• The problems & solutions are, by their nature, rather general, and occur in many different situations in DS.


SECTION 8.1: TIME IN DISTRIBUTED SYSTEMS


Time/Clocks• Physical clocks:

– Problem: Often simply need exact time, not just an ordering.

• Previously solved by time in terms of Sun Transits*

– Solution: Universal Coordinated Time (UTC):

• Based on number of transitions per second of caesium 133 atom**.

• At present, real time is taken as average of ~50 caesium-clocks worldwide.

• Introduces a leap second from time to time to account for fact that days are getting longer (e.g. due to tidal drag, orbital wobbles etc).

• Note: UTC is broadcast through SW radio & satellite. Satellites can give an accuracy of about ±0.5 ms.

CA4006 Lecture Notes (Martin Crane 2018) 14

*Time to reach highest point in sky**Quite accurateLecture 8: Safe Access to Dist’d Shared Resources

Time/Clocks (/2)• Physical clocks:

– Problem

• Suppose have distributed system with a UTC-receiver in it ⇒ we still have to distribute its time to each machine.

– Basic principle

• Each machine has a timer generating interrupt 𝐻 times per second.

• There is a clock in machine 𝑝 that ticks* on each timer interrupt.

• Denote the value of that clock by 𝐶𝑝(𝑡), where 𝑡 is UTC time.

• Ideally, we have that for each machine 𝑝, 𝐶𝑝 𝑡 = 𝑡, or, 𝑑𝐶

𝑑𝑡= 1


*incs s/w clock counting no. of ticks since some (agreed on) time in the past

Lecture 8: Safe Access to Dist’d Shared Resources

Time/Clocks (/3)

• Physical clocks:

• In practice: 1 − 𝜌 ≤𝑑𝐶

𝑑𝑡≤ 1 + 𝜌

𝜌 is the clock’s skew

• From the figure:

– 2 clocks drifting from UTC in opposite

directions in time Δ𝑡 , may be ≤ 2𝜌Δ𝑡 apart

• Goal:

– Don’t let 2 clocks differ by more 𝛿 than time units

=> synchronise every 𝛿/(2𝜌) secs

– 𝛿 termed the rate of drift


Relation btw clock time & UTC when clocks tick at different rates


Time/Clocks (/4)

• Global positioning system

– Basic idea: Can get accurate account of time as side-effect of GPS

– Problem: Assuming satellite clocks are accurate & synchronized:• Takes time before a signal reaches receiver

• Receiver’s clock is definitely out of synch with satellite


Computing a position in a 2D space


Time/Clocks (/5)

𝑥𝑖 , 𝑦𝑖 , 𝑧𝑖

𝑥𝑟 , 𝑦𝑟 , 𝑧𝑟

Measured distance, 𝑑𝑖 = 𝑐(time for light to go from satellite to ship)

So 𝑑𝑖 = 𝑐Δ𝑖

But Δ𝑖 = (𝑇𝑛𝑜𝑤−𝑇𝑖) + Δ𝑟 (𝑇𝑖 is a satellite’s timestamp)

⇒ 𝑑𝑖 = 𝑐Δ𝑖 − 𝑐Δ𝑟(Δ𝑖 is measured time diff,Δ𝑟 is correction for clock deviation)

⇒ 𝑑𝑖= 𝑐Δ𝑖 − 𝑐Δ𝑟 = (𝑥𝑖−𝑥𝑟)2 +(𝑦𝑖 −𝑦𝑟)

2 + (𝑧𝑖−𝑧𝑟)2

i.e. with 4 satellites, now have 4 equations in 4 unknowns


𝑑𝑖

Time/Clocks (/6)• Clock synchronization principles

– Principle I• Every machine asks a time server for accurate time min every 𝛿/(2𝜌) seconds (Network Time Protocol).

• Ok, but must measure round trip delay, incl interrupts & processing incoming messages.

– Principle II• Time server scans all machines periodically, averages, informs each how to

adjust its time wrt. its present time.• Ok, probably get every machine in sync. Needn’t even propagate UTC time.

– Fundamental: Have to take into account that setting time back never allowed ⇒smooth adjustments.


Getting current time from a time server


𝛿/(2𝜌)

Time/Clocks (/7)• Logical Clocks: The Happened-before relationship

– Problem: First must introduce notion of ordering before can order anything.

– The happened-before relation• If 𝑎, 𝑏 are 2 events in same process, 𝑎 comes before 𝑏, then 𝑎 → 𝑏*

• If 𝑎 is the sending of a message, and 𝑏 is the receipt of that message, then 𝑎 → 𝑏

• If 𝑎 → 𝑏 and 𝑏 → 𝑐 , then 𝑎 → 𝑐

– Note: This introduces a partial ordering of events in a system with concurrently operating processes• For such a system, 𝑥 → 𝑦 is not true but neither is 𝑦 → 𝑥

𝑖. 𝑒. 𝑡ℎ𝑒𝑦 ℎ𝑎𝑝𝑝𝑒𝑛 𝑎𝑡 𝑡ℎ𝑒 𝑠𝑎𝑚𝑒 𝑡𝑖𝑚𝑒 (𝑐𝑜𝑛𝑐𝑢𝑟𝑟𝑒𝑛𝑡𝑙𝑦)


*Read: “𝑎 happens before 𝑏”Lecture 8: Safe Access to Dist’d Shared Resources

Time/Clocks (/8)

• Logical Clocks:

– Problem: How to keep a global view on system behaviour that is consistent with the happened-before relation?

(i.e. how can we agree on event ordering?)

– Solution:

– Attach timestamp 𝐶(𝑒) to each event 𝑒, with following properties:

• 𝑃1 If 𝑎 and 𝑏 are two events in the same process, and 𝑎 → 𝑏, then require 𝐶(𝑎) < 𝐶(𝑏).

• 𝑃2 If 𝑎 corresponds to sending a message 𝑚, and 𝑏 to the receipt of that message, then also 𝐶(𝑎) < 𝐶(𝑏).

• Everybody agrees on the values of 𝐶 𝑎 , 𝐶(𝑏).


Time/Clocks (/9)• Logical Clocks: Lamport’s Algorithm

– Problem:– How to attach a timestamp to an event when there’s no global clock?⇒ maintain a consistent set of logical clocks, one per process.

– Solution:– Each process 𝑃𝑖 has local event counter 𝐶𝑖, adjusted as per following rules:

1. For any 2 successive events taking place within 𝑃𝑖, 𝐶𝑖 is incremented by 1.2. Each time a message 𝑚 is sent by process 𝑃𝑖 , the message receives a

timestamp 𝑡𝑠(𝑚) = 𝐶𝑖3. On receipt of message 𝑚 by process 𝑃𝑗, 𝑃𝑗 adjusts its local counter 𝐶𝑗 to

𝑚𝑎𝑥{𝐶𝑗 , 𝑡𝑠(𝑚)} then executes step 1 before passing 𝑚 to the application.

– Notes• Property P1 is satisfied by (1); Property P2 by (2) and (3).• Can still occur that 2 events happen simultaneously. • Avoid this by breaking ties through process IDs.


Note: 𝐶𝑖 is no. of eventsthat have occurred at 𝑖


Time/Clocks (/10)• Logical Clocks: Example

– Impossibility: In (a) 𝑚3 arrives at 𝑃2 before it was sent from 𝑃3– Lamport’s Algorithm:

• 𝑃2 adjusts it’s clock to 1 + sending time (=60) on arrival of 𝑚3 from 𝑃3


Three processes, each with its own clock. The clocks run at different rates

Lamport’s algorithm corrects the clocks.


Time/Clocks (/11)• Logical Clocks:

– Adjustments take place in the middleware layer:


The positioning of Lamport’s logical clocks in distributed systems


Time/Clocks (/12)• Logical Clocks:Example of Totally Ordered Multicast

– Problem:– Sometimes must ensure that concurrent updates on a replicated DB are seen in the

same order everywhere:• P1 adds $100 to an account (initial value: $1000)• P2 increments account by 1% interest in New York

– Two replicas

– Result: In absence of proper synchronization:replica #1 ← $1111, while replica #2 ← $1110.


Updating a replicated database & leaving it in an inconsistent state.


Time/Clocks (/13)

• Logical Clocks: Example Totally Ordered Multicast– Solution:

• Process 𝑃𝑖 sends timestamped message 𝑚𝑠𝑔𝑖 to all others.

• The message itself is put in a local queue 𝑞𝑢𝑒𝑢𝑒𝑖 .

• Any incoming message at 𝑃𝑗 * is queued in queue 𝑗 , according to its timestamp, and acknowledged to every other process.

𝑃𝑗 passes a message 𝑚𝑠𝑔𝑖 to its application if:

(1) 𝑚𝑠𝑔𝑖 is at the head of queue 𝑗

(2) For each process 𝑃𝑘 , there is a message 𝑚𝑠𝑔𝑘 in queue 𝑗 with a larger timestamp. This means that 𝑚𝑠𝑔𝑖 is at the head of 𝑗 ‘s queue and has been acknowledged by other processes.

– Note: We are assuming that communication is reliable & FIFO ordered.


* e.g. acknowledgement.Lecture 8: Safe Access to Dist’d Shared Resources

Time/Clocks (/14)• Logical Clocks: Example

– Observation:• Lamport’s clocks don’t guarantee that if 𝐶(𝑎) < 𝐶(𝑏) that 𝑎 causally

preceded 𝑏

• From diagram, know that for 𝑃2, 𝑇𝑟𝑐𝑣(𝑚1) < 𝑇𝑠𝑛𝑑(𝑚3) but what can be concluded in general from this statement?

• Know 𝑇𝑟𝑐𝑣 𝑚1 , 𝑇𝑠𝑛𝑑(𝑚3) correspond to events that took place at 𝑃2 but also know 𝑇𝑟𝑐𝑣(𝑚1) < 𝑇𝑠𝑛𝑑 𝑚2 but no causality there


Event 𝑎 : 𝑚1 is received in P2 at T = 16;Event 𝑏: 𝑚2 is sent by P3 at T = 20

i.e. clock time is local for unconnectedevents => cannot determine an order


Time/Clocks (/15)• Logical Clocks:

– Problem with Lamport’s Clocks:

• No guarantee that if 𝐶(𝑎) < 𝐶(𝑏) that 𝑎 causally preceded 𝑏

– Solution: Vector Clocks:

• Each process 𝑃𝑖 has an array 𝑉𝐶𝑖 1…𝑛 , where 𝑉𝐶𝑖 𝑗 denotes no. of events that process 𝑃𝑖 knows have taken place at process 𝑃𝑗 .

• When 𝑃𝑖 sends message 𝑚, it adds 1 to 𝑉𝐶𝑖 𝑖 , & sends 𝑉𝐶𝑖 along with 𝑚 as vector timestamp 𝑡𝑠(𝑚).

– Result: on arrival, recipient knows 𝑃𝑖 ’s timestamp (i.e. the number of events at 𝑃𝑖 that causally precede 𝑖)

• When a process 𝑃𝑗 delivers a message 𝑚 that it received from 𝑃𝑖 with

vector timestamp 𝑡𝑠(𝑚) , it

(1) updates each 𝑉𝐶𝑗 𝑘 to max{𝑉𝐶𝑗 𝑘 , 𝑡𝑠(𝑚)[𝑘]}

(2) increments 𝑉𝐶𝑗 𝑗 by 1.

• Put another way, 𝑡𝑠 𝑚 [𝑘] is a tuple consisting of a process’s logical time & its last known time of process 𝑘 in terms of no. of events that occurred at 𝑘

• So with Vector Clocks know that if 𝑉𝐶(𝑎) < 𝑉𝐶(𝑏) ie 𝑎 causally preceded 𝑏CA4006 Lecture Notes (Martin Crane 2018) 28Lecture 8: Safe Access to Dist’d Shared Resources

i.e. max of current value of Vector Clock at 𝑃𝑗for 𝑘 & no. of events at 𝑃𝑘 before 𝑚 sent by 𝑖

Vector Clock Rules


29

• Each process keeps a vector of clock values for the

system

• Initially all clocks are zero.

• If a process has an internal event=> it increments its own logical clock in the vector

• If a process sends a message=> it increments its own logical clock in the vector (but an event causing a

message send only advances the clock once) and sends a copy of its own

vector with message.

• Each time a process receives a message,=> it increments its own logical clock in the vector and updates each element

in its vector by taking the maximum of the value in its own vector clock and

the value in the vector in the received message (for every element).

Vector Clocks Example


Process A Process B

A Clock:A1, B0, C0

Process CA Clock:A0, B0, C0

B Clock:A0, B0, C0

C Clock:A0, B0, C0

Msg(data, A1, B0, C0)

B Clock:A1, B1, C0

Msg(data, A1, B2, C0)B Clock:A1, B2, C0

C Clock:A1, B2, C1

Timestamp

Time/Clocks (/16)• Vector Clocks:

A Digression on Message Timestamps

– If event 𝑎 has timestamp 𝑡𝑠 𝑎 then 𝑡𝑠 𝑎 𝑖 − 1 denotes number of events processed at 𝑃𝑖 that causally precede 𝑎

– Hence, when 𝑃𝑗 gets a message from 𝑃𝑖 timestamped 𝑡𝑠 𝑚 , it knows how many events have occurred at 𝑃𝑖 that causally preceded the sending of 𝑚

– This way, it knows how many events have occurred at other processes prior to the sending of 𝑚 by 𝑃𝑖


Time/Clocks (/17)• Vector Clocks: Causally Ordered Multicasting*

– Observation:

• Can now ensure that a message is delivered only if all causally preceding messages have already been delivered.

• Note, in terms of messages sent and received 𝑉𝐶𝑖 𝑗 = 𝑘 means that 𝑃𝑖knows that 𝑘 events have occurred at 𝑃𝑗

– Adjustment:

• 𝑃𝑖 increments 𝑉𝐶𝑖 𝑖 only on sending a message, & 𝑃𝑗“adjusts” 𝑉𝐶𝑗[𝑘] (to

𝑚𝑎𝑥{𝑉𝐶𝑗 𝑘 , 𝑡𝑠(𝑚)[𝑘]} on receiving a message (i.e., effectively doesn’t

change 𝑉𝐶𝑗[𝑗]).

𝑃𝑗 postpones delivery of 𝑚 until:

• 𝑡𝑠 𝑚 𝑖 = 𝑉𝐶𝑗 𝑖 + 1 (i.e. 𝑚 is next message 𝑃𝑗expects from 𝑃𝑖 )

• 𝑡𝑠 𝑚 𝑘 ≤ 𝑉𝐶𝑗[𝑘] for 𝑘 ≠ 𝑖. (i.e. 𝑃𝑗 has seen all messages seen by 𝑃𝑖when 𝑃𝑖 sent 𝑚)


* Not as strong as Totally Ordered Multicasting.Lecture 8: Safe Access to Dist’d Shared Resources

Time/Clocks (/18)• Vector Clocks: Example 1

– Recall each time message 𝑚 is sent by process 𝑃𝑖, the message receives a timestamp 𝑡𝑠(𝑚) = 𝐶𝑖 (𝐶𝑖denotes no. of events at occurred at 𝑃𝑖)

– Thus when 𝑃𝑗 receives a message from 𝑃𝑖 it knows about the number of

events that have occurred at 𝑃𝑖 before the sending of the message 𝑚.


𝑡𝑠 𝑚 = 1,0,0 ⇒ 𝑉𝐶1 1,1,0𝑡𝑠 𝑚∗ = 1,1,0 ⇒ 𝑉𝐶0 1,1,0

At (𝟏, 𝟎, 𝟎) local time 𝑷𝟎

𝐬𝐞𝐧𝐝𝐬 𝐦𝐞𝐬𝐬𝐚𝐠𝐞 𝒎 𝐭𝐨 𝑷𝟏, 𝑷𝟐

After 𝒎 arrives, 𝑷𝟏

sends 𝒎* to 𝑷𝟎, 𝑷𝟐

𝑷𝟎 delivers 𝒎* cos𝒕𝒔 𝒎∗ = 𝑽𝑪𝟎 𝟏 + 𝟏

Delivery of 𝒎* delayed by 𝑷𝟐

until 𝒎 is received & delivered by 𝑷𝟐 ‘s application layer


Time/Clocks (/19)• Vector Clocks Sample Question:

• Given a system of three processes 𝑃0, 𝑃1, 𝑃2– If the clock in process 2, 𝑉𝐶2 = 0, 2, 2 and process 2 has just

received a message with a timestamp 𝑡𝑠 𝑚 = (1, 3, 0) from 𝑃01. What information does 𝑃2 have?

2. If implementing causally ordered multicasting, what will 𝑃2 do when receiving 𝑚 from 𝑃0 ?


Time/Clocks (/19)• Vector Clocks: Example 2 Three processes 𝑃0, 𝑃1, 𝑃2

– Take 𝑉𝐶2 = 0, 2, 2 & 𝑡𝑠 𝑚 = (1, 3, 0) from 𝑃01. What information does 𝑃2 have?

2. What will it do when receiving 𝑚 from 𝑃0 ?

– 1. aware of 2 events that have taken place at 𝑃1 & 𝑃2 & none at 𝑃0 ; whensent 𝑚,𝑃0 not aware of 2 events at 𝑃2– but that doesn’t affect clock at 𝑃2

– 2. To deliver 𝑚 to 𝑃2 recall rule for Causally Ordered Multicasting:𝑃𝑗 postpones delivery of 𝑚 until:

a) 𝑡𝑠 𝑚 𝑖 = 𝑉𝐶𝑗 𝑖 + 1 (i.e. 𝑚 is next message 𝑃𝑗expects from 𝑃𝑖 )

b) 𝑡𝑠 𝑚 𝑘 ≤ 𝑉𝐶𝑗[𝑘] for 𝑘 ≠ 𝑖. (i.e. 𝑃𝑗 has seen all messages sent by 𝑃𝑖when 𝑃𝑖 sent 𝑚)

– For a) 𝑡𝑠 𝑚 0 = 𝑉𝐶2 0 + 1✓

– For b) 𝑡𝑠 𝑚 1 ≤ 𝑉𝐶2 1 ⇒ 3 ≤ 2 ; 𝑡𝑠 𝑚 2 ≤ 𝑉𝐶2 2 ⇒ 0 ≤ 2 ✓

𝑃2 will adjust 𝑉𝐶2 0 to 1, inc 𝑉𝐶2 2 to 2 & wait for msg from 𝑃1 𝑉𝐶2 = 1, 3, 2 eventually


Documents

LECTURE 7: WEB SERVICES EVENT BASED SYSTEM …rbrennan/CA4006...• Event-Sourcing – Event store as the principle source of truth e.g. git version control – Provides strong audit