Distributed Market Broker Architecture for Resource Aggregation in

Grid Computing Environments

Morihiko Tamai, Naoki Shibata*, Keiichi Yasumoto, and Minoru Ito

Nara Institute of Science and Technology,*Shiga University

Background

• PC Grid and peer-to-peer computing systems (SETI@home, GIMPS and cell computing) – Achieve more than super computer power at

extremely low cost.– Aggregated computing power is used for

public objectives.

• We want to use these computing power for personal objectives– e.g., video encoding

Background (cont.)

• Problem:– Resource providers wouldn’t allow strangers

to use their computers without appropriate incentive

• In Cell computing (by NTT DATA), incentives for PC users are provided by giving them electronic currency.

• But, users can only provide resources. They can’t use other people’s resources.

• Popcorn[4], Java Market[5], JaWS[6], Nimrod-G[7], etc.

• Each user is resource seller or buyer.• Resource seller receives e-money by offering

his/her computational resource.• Resource buyer pays e-money for using

computational resource provided by seller.• Appropriate price of each resource is automatically

decided based on supply and demand.

Market-based Resource Sharing

• Seller wants to find buyer with highest price.• Buyer wants to find seller with lowest price.

• The existing systems utilize a market broker placed in a network node, which mediates between sellers and buyers.

Market-based Resource Sharing (cont.)

Our Goal

• However, in the existing systems…– Market broker is implemented by centralized server,

and with no detailed consideration of its scalability– Can handle only one-to-one order matching

But, if a user wants to use relatively large computation power, it is desirable to match a multiple set of sell-orders to one buy-order.

• Our contributions:– Decentralized market broker architecture.– One-to-many order matching.

Outline

1. Introduction

2. Framework for market-based resource sharing

3. Distributed Market Broker Architecture

4. Experiments and Results

5. Conclusion

Sell/buy Orders• Resource type

– CPU power (MIPS), memory amount (MB), disk amount (GB), occupation time of resource (min)

• Resource vector– r = (r1, r2, …, rn)

– Hereafter, for simplicity, we suppose n = 2.• Sell/buy orders

– Tuple of resource vector and sell/buy price: (r, price)

– e.g., sell-order s = ((1000MIPS, 80MB), 200)

Order Matching

• Matching condition (same as NYSE)– Condition A:

– Condition B: If there are multiple sell/buy orders which satisfy condition A, one with the highest/lowest price is selected for sell/buy orders.

Sell-order: S Buy-order: B

S.r1

S.r2 S.price

B.r1

B.r2 B.price

Order Matching (cont.)

Market brokerSellers buyers

((3, 3), 4) ((2, 1), 2)

Buy price is less than the sell price

Order Matching (cont.)

Market brokerSellers buyers

((3, 3), 4)

((2, 2), 3)

((2, 1), 2)

((1, 1), 5)Match

From condition B, the buy-order matches the second sell-order, because the sell price is lower.

Outline

1. Introduction

2. Framework for market-based resource sharing

3. Distributed Market Broker Architecture

4. Experiments and Results

5. Conclusion

Logical Space for Orders

CPU power0

Mem

ory

am

ount

Sell-order s2 =((80,40), 100)

Sell-order s1 =((60,70), 90)

max

max

Map each order to the point of the two-dimensional logical space

s1 (90)

60

70

s2 (100)

80

40

Logical Space for Orders (cont.)

Sell-order s1 =((60,70), 90)

CPU power0

max

Mem

ory

am

ount

max

Sell-order s2 =((80,40), 100)

s2 (100)

s1 (90)

Buy-order b

Valid region of

sell-order s1

Dividing Order Space

0max

maxm2m1 m3

m5m4 m6

m8m7 m9

• Divide the whole space into sub-regions.• Then, one server node is assigned to each sub-region.

Dividing Order Space (cont.)

0max

maxm2m1 m3

m5m4 m6

m8m7 m9

• When a user registers a new order, he/she sends the order to the one of the server nodes.

s1 (70)

Dividing Order Space (cont.)

0max

maxm2m1 m3

m5m4 m6

m8m7 m9

• When a user registers a new order, he/she sends the order to the one of the server nodes.

• If new sell-order s1 = ((80,40), 70) is sent to m8, it is forwarded to m6 using CAN routing [11].

• Finally, s1 is retained by m6. s1 (70)

s1 (70)

Dividing Order Space (cont.)

0max

m2m1 m3

m5m4 m6

m8m7 m9

Problem: s1 and b can’t match even though they satisfy matching condition, because they are retained by different server nodes.

Buy-order b (75)

s1 (70)

Order Replication

0max

m2m1 m3

m5m4 m6

m8m7 m9

• Make appropriate server nodes hold replicas of the order.

s1 (70)

Problem: s1 and b can’t match even though they satisfy matching condition, because they are retained by different server nodes.

Order Replication (cont.)

0max

m2m1 m3

m5m4 m6

m8m7 m9

s1 (70)• Make appropriate server nodes hold replicas of the order.• Use flooding based delivery method to send replication messages.

Problem: s1 and b can’t match even though they satisfy matching condition, because they are retained by different server nodes.

Order Replication (cont.)

0max

s2 (80)

s3

s4

s5

maxm2m1 m3

m5m4 m6

m8m7 m9

s1 (70)

Problem: server nodes on the left below are overloaded to retain replicas.

Order Replication (cont.)

0max

max

s3

m2m1 m3

m5m4 m6

m8m7 m9

s2 (80) s1 (70)• pay attention to sell price of sell-orders.• we can realize that m7 and m8 do not need to retain replicas of s2, because s2.price > s1.price

Problem: server nodes on the left below are overloaded to retain replicas.

Buy-order b (75)

One-to-many Order Matching

• We suppose there are n sub-tasks.

• Let {b1, b2, …, bn} denotes the set of buy-orders for the sub-tasks. We call each bi sub-buy-order.

• We suppose the total budget is specified by user to buy resources for n sub-tasks.

• We want to match all sub-buy-orders simultaneously.

One-to-many Order Matching (simple method)

First, we think about a simple method.

0 max

maxm2m1 m3

m5m4

m8m7 m9

s2 (80) s1 (70)

s3 (100)

b1b2

• two sub-tasks {b1, b2}.• total budget is 150.• In simple method, we register b1, b2 as independent buy-orders.• Problem: how to specify buy prices of the buy-orders?

One-to-many Order Matching (simple method)

First, we think about a simple method.

0 max

maxm2m1 m3

m5m4

m8m7 m9

s2 (80) s1 (70)

s3 (100)

b1b2

• two sub-tasks {b1, b2}.• total budget is 150.• In simple method, we register b1, b2 as independent buy-orders.• Problem: how to specify buy prices of the buy-orders?

(80)(70)

One-to-many Order Matching (simple method)

First, we think about a simple method.

0 max

maxm2m1 m3

m5m4

m8m7 m9

s2 (80) s1 (70)

s3 (100)

b1b2

• two sub-tasks {b1, b2}.• total budget is 150.• In simple method, we register b1, b2 as independent buy-orders.• Problem: how to specify buy prices of the buy-orders?

(75)(75)

One-to-many Order Matching (our method)

0 max

m2m1 m3

m5m4

m8m7 m9

s2 (80) s1 (70)

s3 (100)

b1b2

• We allow buyers to specify wild card as buy price.• Each server node finds a sell-order with the lowest price which satisfies matching condition. • Then, each server node sends the buyer tentative match message.

( * )( * )

buyer

One-to-many Order Matching (our method)

0 max

m2m1 m3

m5m4

m8m7 m9

s2 (80) s1 (70)

s3 (100)

b1b2

• Then, buyer checks if the total budget is enough to buy all sell-orders.• If so, the buyer sends confirmation messages.• Finally, server nodes confirm the one-to-many order matching.

( * )( * )

buyer

Budget >= 150 ?

Outline

1. Introduction

2. Framework for market-based resource sharing

3. Distributed Market Broker Architecture

4. Experiments and Results

5. Conclusion

Experiment 1

• To evaluate what extent our distributed architecture can mitigate each server load.

• Settings:– 15000 of sell-orders and buy-orders are generate

d.– The number of resource types: 2.– The amount to sell or buy each resource: unifo

rm random values in [0:100].– Sell/buy price: Gaussian distribution whose mea

n and standard deviation are (v1+v2) and 30. (Here, vi denotes the amount of i-th resource)

Experiment 1 (cont.)

• We measured average number of messages received per node.

• The messages are categorized into three:(1) buy/sell orders sent from clients.

(2) replication messages for orders.

(3) deletion messages for replicas.

Overhead due to decentralization of server nodes

ScalabilityA

vera

ge

Nu

mb

er

of M

ess

ag

es

rece

ive

d p

er

serv

er n

ode

• The number of messages received by each server node can be reduced significantly by increasing the number of server nodes.• The load is balanced among server nodes uniformly.

Experiment 2• To evaluate the difference of matching results

between our method and centralized server node.• Users’ satisfaction degree of matching result:

– When a new order o1 matches an order o2 which is already retained by server node,

• D = o1.buyPrice – o2.sellPrice, if o1 is buy-order.• D = o2.buyPrice – o1.sellPrice, if o1 is sell-order.

– In the case of centralized server node, o2 which maximizes D is always selected for o1.

• DG: value of D when using centralized server node.– In the case of decentralized server node, value of D is

equal or less than DG because there are transmission delay among server nodes.

– Define users’ satisfaction degree of matching result as DL / DG.

• DL: value of D when using decentralized server nodes.

Difference of matching results compared with centralized server node

The case of centralized server node

Our architecture achieves almost the same satisfaction degrees (>= 0.95) as the case of centralized server node

Transmission delay among server nodes (ms)

Experiment 2

• To evaluate effectiveness of one-to-many order matching method.

• We measured probabilities of successful one-to-many matching using proposed method, and compared it to the case of simple method.

• Simple method:– We generated n buy-orders using the same settings

of the first experiment, and registered these orders as independent orders (n is determined at random according to exponential distribution whose mean is 10).

– Matching fails if there is a buy-order which is not matched to sell-order until the end of simulation time.

• Our one-to-many matching method:– We used the same buy orders as simple method.– The total budget is the sum of buy prices of n buy-

orders.

Experiment 2 (cont.)

Efficiency of one-to-many order matching

Our method achieves higher matching probability than the simple method.

Conclusion

• Our architecture…– can significantly reduce the number of

messages received by each server node.– greatly increases the availability of resources

using one-to-many matching method.

• Future work…– Dynamic division method of each sub-region

assigned to a server node will be treated.

Thanks

Introduction of Time Slot

0max

m2m1 m3

m5m4 m6

m8m7 m9

Problem: when a large number of orders are arrived at one server node in a short time period, the server node may be overloaded due to transmitting replication messages.

s1 (70)After T

After T• Aggregate replication messages during every time interval T.

Order Deletion

0max

maxm2m1 m3

m5m4 m6

m8m7 m9

• Problem: s1 matches to b1 and b2.• To keep consistency, when a server node matches an order to a replica, let the server node check if the original order exists or not.• If original exists, deletes the original.• If not, the matching is canceled.

s3 (90)

s1 (70)s2 (80)

Buy-order b1 (75)

Buy-order b2 (75)

One-to-many Order Matching (cont.)

0max

maxm2m1 m3

m5m4

m8m7 m9

s2 (80) s1 (70)

• When a server node receives sub-buy-order, it finds a sell-order with the lowest price which satisfies matching condition, and sends the buyer tentative match message.

s3 (100)

b1b2

buyer

One-to-many Order Matching (cont.)

0max

maxm2m1 m3

m5m4

m8m7 m9

s2 (80) s1 (70)

• Buyer checks whether the total budget is enough to buy all sell-orders.• If so, the buyer sends confirmation messages.

s3 (100)

b1b2

buyer

One-to-many Order Matching (cont.)

0max

m2m1 m3

m5m4

m8m7 m9

s2 (80) s1 (70)

• Each server node checks whether the sell-order still exists.• If so, server node changes the state of sell-order to intermediate state, and sends ack message. • Otherwise, it sends nack message.

s3 (100)

b1b2

buyerack

One-to-many Order Matching (cont.)

0max

maxm2m1 m3

m5m4

m8m7 m9

s2 (80) s1 (70)

• If the buyer receives ack messages from all server nodes, the sub-buy-orders finally matches to the sell-orders, and buyer sends commitment messages. s3 (100)

b1b2

buyer

b1 = ((40,50), undefined). b2 = ((70,40), undefined).

commit

One-to-many Order Matching (cont.)

0max

maxm2m1 m3

m5m4

m8m7 m9

s2 (80) s1 (70)

• Otherwise, if at lease one nack message received, buyer sends cancel messages to free intermediate state of sell-orders. s3 (100)

b1b2

buyer

b1 = ((40,50), undefined). b2 = ((70,40), undefined).

cancel

More Detailed Settings for the Experiment 3

• Orders generated:– 15000 sell-orders– 13500 buy-orders– 1500 buy-orders which include n buy-orders

(Here, n is determined at random according to exponential distribution whose average is 10)

(1000)

(100)