View
243
Download
1
Category
Preview:
Citation preview
1
A Unified Modeling Framework for Distributed Resource Allocation of
General Fork and Join Processing Networks
in ACM SIGMETRICS 2010
2
Agenda
• Introduction• System Description• A Unified Modeling Framework• Distributed Algorithm• Performance Analysis• Extensions• Evaluation• Conclusion
3
Introduction
• Fork and Join Processing Networks arise in many networked or distributed computing areas, including cloud computing.
• Data processing software typically consists of a rich set of semantics that define– Join: how data from various input links are processed– Fork: how output data are disseminated to each output link
5
Introduction (cont’d)
• Four types of fork and join:– synchronous -join, asynchronous -join • With a synchronous join, data from all input buffers are
assembled simultaneously; whereas an asynchronous join processes data from one input buffer at a time.
– synchronous -fork, asynchronous -fork• A synchronous fork dis-assemble or duplicate data over
all output links simultaneously; while an asynchronous fork sends data over one output link at a time.
6
Introduction (cont’d)
• Our goal is to address the problem of distributed resource allocation in such general fork and join processing networks.– how to formulate the resource allocation problem for a
general fork and join processing network? – the different types of semantics and processing
requirements introduce complex interdependencies among the various data flows
– These ingredients make the distributed resource allocation problem much more challenging.
7
System Description• G0 = (V0, E0), where V0 = S0P0D0 consists of three types of nodes: (data)
sources S0, (processing) tasks P0 and (data) sinks D0.• Assume there is a finite set of resources R. We view each resource as a
‘server’ rR, which has limited capacity Rr.• Assume the tasks have been assigned to the various servers
Objective: maximize the sum of the utilities for all the flow output rates
8
A Unified Modeling Framework
• One important characteristic of a fork and join processing network is that it can be “lossy”.
• In such a “lossy” network, the flow constraint is that a child node cannot consume more than what the parent node can output, i.e., we must have
12
The Dual Decomposition (cont’d)
• When the multipliers (pm,mM) are given, the solution to the above congestion control and resource allocation problems are straight-forward:
13
Distributed Algorithm
• Shadow Queue Based Algorithm– Since utility is charged at the sinks, update equation (13)
suggests that the shadow prices (multipliers) can be related to credits created at the sinks and that flow bottom up towards the sources.
– That is, a back-pressure algorithm can be applied to the shadow queues on the reversed graph
– Credits in the shadow queues can be used to guide the processing of the real queues
16
Evaluation
• We use the task graph in Figure 1.• All resource constraints are set to 100 except Rr1 = 90.• The utility functions are set to the
natural log function (Ua(x) = ln(x)).
17
Evaluation (cont’d)
• After 1000 iterations, the resource for server r2 is reduced from 100 to 70.
• After another 1000 iterations, the importance of the d2 output stream is increased by changing its utility function from ln(x) to 2 ln(x).
19
Other detail results
Real queue at sink is stable. P5 is a simultaneous join and is involved in a lossy link from p3.
20
Conclusion
• This paper propose a unified modeling framework that can represent all combinations of fork and join semantics, and formulate the resource allocation problem into an elegant optimization problem
• The authors further present a shadow queue based algorithm to solve the resource allocation problem in a distributed fashion.
21
Comments
• This paper considered a more complex task network and resource allocation problem.– Flaw: tasks on which server is fixed.
• A distributed resource allocation algorithm would be a major concern in the research of cloud computing.– The approach in this paper is still too complex.
• The issue of service model of a cloud service must also be considered.
Recommended