1

Inter-Datacenter Bulk Transfers:Trends and Challenges

Long Luo, Hongfang Yu, Klaus-Tycho Foerster, Max Noormohammadpour, and Stefan Schmid

Abstract—Many modern Internet services and applicationsoperate on top of geographically distributed datacenters, whichreplicate large volumes of business data and other content, toimprove performance and reliability. This leads to high volumesof continuous network traffic between datacenters, constitutingthe bulk of traffic exchanged over the Wide-Area Networks(WANs) that connect datacenters. Operators heavily rely on theefficient and timely delivery of such traffic, as it is key forthe performance of distributed datacenter applications, and bulkinter-datacenter traffic also plays an important role for networkcapacity planning.

In this paper, we discuss the unique and salient features of bulkacross-datacenter transfers and extensively review recent tech-nologies and existing research solutions proposed in the literaturefor optimization of such traffic. Moreover, we discuss severalchallenges and interesting directions that call for substantialfuture research efforts.

I. INTRODUCTION

Datacenter networks are growing explosively in both sizeand numbers with the increasing popularity of online ser-vices on many fronts (e.g., health, business, streaming, socialnetworking)—they have become a critical infrastructure oftoday’s digital society. The total number of hyperscale data-centers that are capable to scale out on-demand has increasedto over 500 in 2019, which has been tripled since 2013 [1].It takes only two years to build more than 100 hyperscaledatacenters, and the rate is accelerating. To offer cloud serviceswith improved performance, providers typically build datacen-ters in distant regions close to their global customers, see Fig 1.For example as of 2019, Amazon, Microsoft, and Googleoperate datacenters across dozens of geographical regions andmore than one hundred cities.

With these geographically dispersed datacenters, cloudproviders can easily support thousands of services such as weband streaming. Operating services globally, in turn, increasesthe inter-DC WAN pressure in that services continuouslyproduce an increasing amount of data transfers that typicallyrange from tens of terabytes to petabytes among their workingdatacenters for increasingly stringent availability, performance,and consistency [2]–[5]. For example, Facebook is experi-encing an exponential growth of the traffic carried by theirinter-DC networks over the years, this trend is expected tocontinue in the future [6]. A relatively small portion (e.g., 5%-15% [4]) of inter-DC traffic is user-facing (e.g., interactive

Long Luo is with the University of Electronic Science and Technologyof China; Hongfang Yu is with the University of Electronic Science andTechnology of China and Peng Cheng Laboratory; Stefan Schmid and Klaus-T. Foerster are at the Faculty of Computer Science, University of Vienna; MaxNoormohammadpour is with Facebook. Corresponding author: Hongfang Yu.

user search queries) which happens between customers anddatacenters. User-facing traffic is highly sensitive to delay andhence always prioritized over application to application trafficand delivered with appropriate reserved bandwidth. On theother hand, the bulk of traffic carried by inter-DC networks isfast-growing internal traffic, which is application to application(e.g., search index synchronization and database replication).Internal traffic can tolerate additional delay and is hencedelivered over the leftover bandwidth of high-priority user-facing traffic.

In order to support massive volumes of traffic, inter-DCnetwork operators, such as Amazon and Facebook, need toinvest in expensive high capacity WAN backbones that connecttheir datacenters globally. Moreover, many cloud providersinvest large sums of capital in leasing bandwidth from InternetService Providers (ISPs) to transfer data across their data-centers. In general, major cloud providers spend in the orderof hundreds of millions of dollars per year (amortized) onmaintaining such connectivity. Therefore, efficient utilizationof network bandwidth is critical to maximize cost savings andoptimize performance given existing capacity at any giventime. Motivated by these trends, the networking communityhas recently done many efforts on the design of new trafficengineering solutions for inter-DC traffic, especially to deliverbulk transfers efficiently. To this end, their solutions pursueresearch goals of improving resource utilization [2], [3], [7],meeting transfer deadlines [4], [8], [9], minimizing transfercompletion times [10], reducing transmission costs [11], etc.

Our short tutorial paper provides an introduction to currenttrends and open problems in this area. We discuss communi-cation patterns, performance metrics, and objectives of inter-datacenter bulk transfers. Moreover, we review the transfertechniques used to optimize the large inter-DC transfers inexisting solutions and point selected open problems in theliterature that will be of interest for future work.

II. CURRENT SITUATION OF INTER-DC TRANSFERS

We first briefly review the inter-DC bulk transfers studied bycurrent works, considering three dimensions: communicationpatterns, performance metrics, and operator objectives.

A. Communication Patterns

Inter-DC bulk transfers can be broadly classified into thefollowing communication patterns, with respect to the numberof destinations on a data transfer, see Fig. 1. One is the Pointto Point (P2P) transfers that deliver data from the sourcedatacenter to a single destination datacenter. For example, to

2

Datacenter Location

Cloud Services

Datacenter

Service data P2P transfer P2MP transfer

Fig. 1: Illustration of an inter-datacenter WAN and bulktransfers, as in Google’s B4 [3].

protect a datacenter against failures or natural disasters suchas snowstorms and floods, cloud backup services may back updata onto a geographically distant datacenter. Quite a few ef-forts are centered around P2P transfers such as Google B4 [3],Microsoft TEMPUS [5], and Amoeba [4]. Another one isthe Point to Multipoint (P2MP) transfers which replicate datafrom the source datacenter to multiple destination datacenters.Geo-replication is usually the main datacenter service thatgenerates transfers with such communication patterns [10].For example, search engines use geo-replication to periodically(e.g., every 24 hours) synchronize search index updates acrosstheir datacenters, video streaming services disseminate high-definition video content to multiple locations close to theirregional customers [4], and financial institutions back up theirdaily transaction records to datacenter sites [12].

P2MP transfers is forming a large portion of workloadsover inter-DC WANs (e.g., up to 91% for some large-scaleonline service providers [13]). Hence, many of the currentoptimizations on the basis of P2P transmission are becomingless efficient. In this past two years, we have seen an increasingnumber of efforts (e.g., [10] and [9]) devoted to optimizingP2MP transmission. This trend is likely to continue: geo-replication traffic will continuously grow with the rapid ex-pansion of datacenters and the proliferation of global cloudservices, which demands more effort dedicated to P2MPtransfers to improve efficiency.

B. Performance Metrics

Inter-DC bulk transfers typically have a size that rangesfrom tens of terabytes to petabytes, and a key performancerequirement of them is timely delivery [5]. Delayed comple-tion will significantly degrade service quality and even violateSLAs between users and service providers. The performancemetrics considered by a majority of prior work on inter-DClarge transfers can be roughly classified into two types: 1) tomeet deadlines and 2) to minimize transfer completion times.

Deadlines specify the time period that the data deliveryneeds to be finished, which is imposed by many cloudservices [4]. Such deadlines may represent, e.g., differentconsumer SLAs, or the importance of transfers for businessesand organizations. Most prior works (e.g., [4], [8], [9]) focus

TABLE I: Representative transfer scheduling objectives

Objective DescriptionFairness Resources should be fairly allocated among

transfers to achieve max-min fairness [2], [3] orcompletion fairness [5].

MaximizingUtilization

Fully using available network bandwidth is nec-essary to improve network throughput [2], [3].

Minimizing TransferCompletion Times

Short completion times improve the overall qual-ity of services and network efficiency [10].

MaximizeSatisfied Demand

For deadline-constrained transfers of equalvalue, maximizing the number of deadline-satisfied transfers maximizes the satisfied de-mand and the network utility [4], [8].

MinimizingTransmission Cost

Large data transmissions over the inter-DC net-works of service providers are of substantialcost. A fundamental objective is hence to re-duce or even minimize the transmission chargesincurred by the inter-DC bulk transfers [11].

on deadlines, where a transfer may be delayed in favor ofanother one with a closer deadline. Recent work (e.g., [4]) alsoconsiders transfers associated with different types of deadlines,i.e., hard and soft deadlines. In the case of hard deadlines,a late transfer is useless to applications and so guaranteeingcompletion prior to the deadline is necessary to improve thetransfer efficiency. Transfers with soft deadlines, on the otherhand, can be delayed to some extent given a decreasing utilityover time, but still need to finish before some later deadline.

Transfer completion time refers to the total time needed tofinish a transfer request from when it arrives at the network.As delivering data as soon as possible is critical to the perfor-mance of many global-scale cloud services, completion timeis the primary metric to be considered when applications donot specify a deadline. Most work on deadline-unconstrainedbulk transfers focuses on the average completion time [10],[12], while a handful of them (e.g., Owan [12]) also considerthe total time to complete a given collection of transfers, i.e.,the makespan.

Performance metrics of bulk transfers can be used to formconstraints on data delivery and various objective functions,as outlined in the following section, depending on what isimportant to service providers and applications.

C. Objectives of Scheduling Inter-DC Bulk Transfers

Table I summarizes the common objectives of prior trafficengineering solutions for inter-DC bulk transfers. Fairness isone of the main objectives of the solutions [2], [3], [5] con-sidered by large technological companies, such as Microsoftand Google. In particular, they focus on max-min (throughput)fairness [2], [3] and completion fairness that maximizes theminimal deadline-meeting fraction when it is not possible tomeet all deadlines for transfers with soft deadlines [5].

For many large cloud providers which deploy their owndedicated WANs connecting distant datacenters, most of theirinter-DC WAN resources and costs are fixed over relativelyshort periods of time. As a result, they also focus on max-imizing the resource utilization by accommodating as manydata transfers as possible [2], [3]. As timely delivery is crucialto the performance of many online services, minimizing theaverage, median, or tail transfer completion is another goalfor many providers [10], [12]. For deadline-sensitive bulk

3

Route, RateScheduling,TopologyNetwork

Status

Transfer requests

Data Center Inter-Datacenter LinkSDN Controller

Fig. 2: High-level overview of an SDN-based control systemfor engineering bulk transfers over inter-DC WANs.

transfers, delivering traffic without paying attention to dead-lines could waste precious inter-DC link bandwidth, as latetransfers can have much lower utility. In such cases, the goalof minimizing the fraction of missed deadlines or latenesscan maximize the value of deadline-sensitive data transfers(satisfied demands) and improve the effective utilization ofscarce and expensive WAN resources [4], [8], [9].

On the other hand, cloud service providers which buildtheir private inter-DC WANs often invest huge on buildingown optics or leasing links from ISPs. One major objective ofthese providers is to minimize the cost incurred by data trans-missions. Hence, many prior works (e.g., TrafficShaper [11])consider the intensity of inter-DC bulk transfers and aim tominimize the transmission cost of the network bandwidthaccording to certain usage-based charging models.

III. TECHNIQUES FOR BULK TRANSFERS

In this section, we first introduce a Software Defined Net-working (SDN) based control system that is widely adoptedby many solutions for bulk transfer optimization. Then, wewill review the major techniques used by prior work in theliterature, which rely on bulk transfer traffic characteristics.

A. SDN-based Control Systems

An increasing number of companies and organizations,such as Google [3], Microsoft [2], and Facebook [6], haveadopted SDN to operate their inter-DC WANs in the pastfew years. The reasons are twofold: first, an inter-DC WANusually has only a few dozens of datacenters, which any SDN-style centralized network control can easily manage withoutscalability issues. Second, it’s the flexibilities and optimizationopportunities of SDN, which enhance the network abilityto optimize the WAN resources allocation to increase theoverall network utility. Fig. 2 provides a high-level overview ofSDN-based control systems for inter-DC bulk transfers, wherecloud services submit their transfer requests and the providersoptimize the delivery (e.g., forwarding routes, transmissionrates and scheduling policies) of these requests and even

the network topology towards certain objectives, with well-designed transfer allocation approaches and techniques.

In the following, we provide an overview of several selectedtechniques used in prior research and discuss their respectivecontributions.

B. Transmission Techniques

Generally, transmission control techniques determine therouting and forwarding of traffic, the rate at which trafficis transmitted, and the delivery scheme of data that needsto traverse the inter-DC WAN. In the case of inter-DC bulktraffic, these techniques largely rely on the characteristics ofsuch traffic and are usually tailored to scheduling objectivesconcerned by service providers. A “key characteristics” [4]are associated deadlines (timely delivery requirement), but atthe same time, such transfers are delay-tolerant (i.e., “can beserved more flexibly” [5]). The combination of both henceenables scheduling techniques, to be also covered in moredetail later.

1) Routing and forwarding: Routing schemes decide whichforwarding route, including intermediate datacenters and inter-DC links, the traffic takes when traversing the inter-DC WANfrom the source towards the destination datacenter(s). With theadoption of SDN, routing decisions are naturally performedin a centralized manner, which allows for (close to) optimaland computationally tractable solutions, as most inter-DCWANs only contain about a few dozen sites [2], [3], [6].Notwithstanding, multiple controllers can be incorporated aswell [14]. The design of centralized routing schemes typicallytake into account whether the routes change in the process ofdata delivery and how the data is delivered.

In this context, routing can be static or dynamic. Staticapproaches keep routes fixed after the initial assignmentand do not update it throughout the delivery process. Staticforwarding paths are convenient for operators, as they avoidthe transient routing inconsistency, such as blackholes andforwarding loops [2], which could arise due to dynamicupdates. Many prior work [4], [5] precomputes a collectionof paths (e.g., k-shortest paths) between every pair of data-centers and uses them as tunnels to deliver bulk traffic amongdatacenters. To adapt to changes in network condition (e.g., theincrease or decrease of traffic intensity, capacity changes, etc.),some research work, such as [9], dynamically recalculatesforwarding routes for every transfer request, either periodically(e.g., every several minutes) or when new transfer requestsarrive, to improve efficiency and performance. Despite thenetwork having to suffer from transient throughput drops forperforming consistent and congestion-free routing updates [2],dynamic routing is still suitable for bulk transfers because suchtraffic is often long-lived and less sensitive to delay.

Routing of bulk transfers can be unicasting or multicast-ing. Unicasting delivers every copy of data over a separatepath every destination, while multicasting can deliver data tomultiple destinations simultaneously. It is natural to delivera P2P transfer via unicasting as leveraged by many existingworks [2]–[5]. For P2MP transfers, some works transformthe setting into multiple P2P transfers, each with one of the

4

Transfer Techniques

Transmission

Scheduling Rate Allocation

Preemption Non-Preemption

End to End Store and Forward

Policies First Come First Serve

Shortest Remaining Processing First

Max-Min Fairness

Spatial Temporal

Admission Control Delivery Scheme

Static

Multipathing

Dynamic MulticastingUnicasting

Steiner Tree Cohorts

Routingand Forwarding

Topology Design

Reservation Rescheduling

Handling Mispredictions

and Failures

Fig. 3: High-level overview of current techniques on bulk transfers

destinations, and apply unicast routing separately to these P2Ptransfers [8]. Meanwhile, several works consider multicast net-works of SDN capabilities, where they deliver P2MP transfersover Steiner trees originating from the source site, flowingthrough or ending with the destination sites [9], [10].

Furthermore, in order to fully use scattered network re-sources, multiple routes, i.e., k-paths (Steiner trees), can alsobe introduced to deliver a single P2P (P2MP) transfer, referredto as multipathing (multicast tree cohorts).

A popular SDN solution of routing implementation is Open-Flow using which highly sophisticated forwarding patternscan be programmed into network elements. For path-basedunicasting, SWAN [2] identifies admissible tunnel paths bylabeling, where VLAN IDs are used as labels to determinewhich tunnel is traversed by every data packet. For tree basedmulticasting, QuickCast [10] and DaRTree [9] use the grouptables in OpenFlow switches to forward a copy of data tomultiple outgoing ports.

2) Rate allocation: Allocating appropriate transmissionrates on the available routes is also crucial to improve theefficiency of bulk transfers over Inter-DC WANs. Herein,transfer rates are flexible and adaptive to network conditionssuch as traffic load and topology.

Rate allocation typically exploits flexibility in dimensions ofspace and time. As network bandwidth differs from locationto location, it is common practice to employ multiple routesfor a single transfer and distribute traffic among these routesto fully use link bandwidth across different spatial locations.Some rate allocation solutions determine the sending rateon each forwarding route, while some plan the total rateon all admissible routes, as well as a splitting ratio amongthese routes. Because high-priority interactive traffic typicallyfluctuates temporally, the leftover bandwidth that can used bybulk transfers accordingly varies over time. Many approaches(e.g., [4], [5]) consider such temporal differences and proposeto allocate time-varying transmission rates for bulk transfers.For example, most solutions use a time-slotted system andchange flow rates across timeslots. The spatial-temporal dy-namics of network resources and the properties (e.g., deadline)of bulk transfers are often combined to enable optimizations.

3) Delivery schemes: The storage capability of intermediatedatacenters may also be exploited to assist bulk transfers,which results in two delivery schemes: End to End (E2E)and Store and Forward (SnF). E2E delivery does not rely onstorage capability of middle datacenters but requires concur-rent availability of nodes and links on transmission routes.Many approaches [2]–[5], [8]–[10] adopt this scheme to avoidthe extra storage cost at intermediate datacenters and highlycomplex network algorithms. With this delivery scheme, theproviders only determine routes and flow rates per given routefor each transfer. To this end, some solution uses a timeexpansion graph. We refer to Fig. 4a, where a time expansiongraph is built for a P2P transfer from DC1 to DC4 across twotimeslots of t1 and t2. This transfer uses spatially differentpath routes, marked using the orange lines in Fig. 4a.

In contrast, SnF delivery allows intermediate datacenters totemporarily store data before delivering it to the next one.Some solutions (e.g., [7]) apply SnF when some inter-DClinks are temporally congested, forwarding data at a latertime when these links are less congested. With SnF, the timeexpansion graph of the P2P transfer in Fig. 4a accordinglyadapts to the introduction of storage: adding admissible linksbetween the snapshots of the same datacenter at differenttimes, as the blue dotted lines in Fig. 4b. As a result, thisrequest from DC1 to DC4 can traverse a path with temporalstorage at datacenter DC3, as in the green dotted line inFig. 4b. Compared to E2E, SnF introduces storage as anadditional dimension for optimization, but storing data atintermediate datacenters incurs additional costs. As the useof data storage, routing, and rate allocation must be carefullycoordinated, SnF transfer allocations can also lead to morecomplex optimization formulations.

C. Scheduling Techniques

As many transfer requests typically coexist in a network,various scheduling techniques can be used to improve theoverall performance by arranging how these transfers utilizethe network resources. We briefly discuss the topics of policies,preemption, reservation, and rescheduling next.

1) Policies: Scheduling policies is to determine the pro-cessing order of coexisting transfer requests. Most solutions

5

t1

t2

DC1

DC1 DC2

DC4

DC3 DC4

DC1 DC2

DC3 DC4

(a)

t1

t2

DC1

DC1 DC2

DC4

DC3 DC4

DC1 DC2

DC3 DC4

(b)

Fig. 4: Time expansion graph models of (a) End-to-End (E2E)and (b) Store-and-Forward (SnF) delivery schemes.

(e.g., [2], [4]) consider transfers with the same weight (pri-ority) and schedule them according to a First Come FirstServed (FCFS) policy. FCFS processes transfer requests inthe order of arrival, which is simple to implement. There arealso a few attempts to use other scheduling policies, suchas Shortest Remaining Processing Time (SRPT) and Max-Min Fairness (MMF), to improve transfer efficiency (i.e.,minimizing completion time [10]). SRPT schedules transfersin an order of remaining completion time, which is optimalfor minimizing mean completion times for P2P transfers overa single link. With P2MP transfers, in particular scenarios,MMF can beat SRPT in minimizing mean completion timesby a large factor when many multicast trees share a link. Ingeneral, it is also possible to apply multiple scheduling policiesin a hierarchical fashion to obtain sophisticated schedulingobjectives. For example, one can classify transfers into twopriorities of low and high and apply SRPT and FCFS acrossthese two classes and within each class, respectively.

2) Preemption: Transfer requests can also be scheduledwith support for preemption or without it, according to whethertheir resource allocations can be revoked. Specifically, non-preemptive scheduling does not allow interrupting or revokingtransfer requests that are being delivered in the network, wherethe resources allocated to a transfer can only be released andreclaimed after completion. Some solutions, such as TEM-PUS [5] and DaRTree [9], adopt non-preemptive schedulingfor bulk transfers with deadlines. In contrast, preemption maytemporarily interrupt an ongoing transfer without coordination,which usually occurs when competing for resources withhigher priority traffic (e.g., interactive traffic). It can also bespecifically designed for scheduling transfers with equal prior-ity to improve efficiency. For example, to reduce the averagetransfer completion time, preemptive scheduling can be used tointerrupt a large slow transfer to schedule more small but fastones. Compared to the non-preemptive that only schedules newtransfers, preemptive scheduling incurs additional complexityand cost, as it needs to maintain the status of existing and newtransfers as a reference for strategies.

3) Reservation: Reserving bandwidth for transfer requestsis an important technique to offer predictable and guaran-teed performance services. For example, the works in [4],[8] guarantee completion before deadlines for accepted bulktransfers by reserving bandwidth into future timeslots. Thereserved bandwidth of a transfer cannot be preempted by other

transfers, but may vary over time, as long as the promisedservice quality (e.g., deadline) is ensured. As a result, a datatransfer may be set to temporarily stop delivering data whenno bandwidth reserved for it at that time—in contrast to non-preemptive scheduling, which does not allow transmissions tobe interrupted before completion.

4) Rescheduling: The bulk transfer problems are oftenin an online setting where user requests are submitted tothe system over time. The optimal allocation decision thatwas made earlier may become outdated and sub-optimal,due to new unforeseen transfer requests arriving at a latertime. Rescheduling existing transfers together with new onescan often improve performance, but at the cost of rapidlyincreasing computational complexity, especially for the systemthat needs to plan resource allocations over a long-term time toguarantee transfer deadlines. To reduce such additional com-putational complexity, selective rescheduling can be performedto only include a few existing transfer requests in a jointrescheduling [4].

D. Other Techniques

1) Admission control: It is necessary to apply admissioncontrol to selectively admit a subset of transfer requestssubmitted to the network out of consideration of guaranteeingtransfer deadlines or maximizing total profit. For example, inbulk transfers with a hard deadline, missing their deadlinemakes the data lose its utility. As a result, most deadline-aware solutions, e.g., [4], [8], [9], only accept transfers that canfinish before deadline and analogously reject the transfers thatwill miss their deadlines. To this end, these solutions use non-preemptive scheduling and reserve the bandwidth to deliveradmitted requests.

2) Topology control: With the emergence of reconfigurablenetwork technologies, the topology of inter-DC WANs canbe quickly reconfigured at the physical layer. A few so-lutions [9], [12] optimize the topology together with thetransfer allocation, where they use cross-layer optimizationalgorithms to jointly design the optical topology and therate allocation for bulk transfers. Although performance couldincrease noteworthily, cross-layer joint optimization inevitablyincreases problem complexity and computational cost.

3) Handling mispredictions and failures: Any transmissionscheme has to deal with dynamic network conditions, espe-cially mispredictions and failures. As bulk transfers can onlyuse the leftover bandwidth of high-priority interactive traffic,the accurate estimation of interactive traffic is beneficial toefficient transfer allocations. Prior works [2], [5] show thatinteractive traffic is predictable in periodic patterns within ashort time period (e.g., 5 minutes). However, mispredictionis nonetheless inevitable. For example, Amoeba [4] observesthat predication accuracy decreases with forecast length. Ac-cordingly, to handle such misprediction, Amoeba sets asidedifferent headrooms for different time slots proportional totemporal distance. In case of failure events, existing solutions(i.e., [4]) recalculate resource allocation decisions, but timestill matters for bulk transfers.

6

IV. OPEN PROBLEMS

In this section, we point out seven open problems that weconsider of special interest for future research. We categorizethese problems into optimizing inter-DC bulk transfers for bet-ter traffic prediction and measurement, improved schedulingpolicies and algorithms, and advanced network techniques.

We note that the challenges for intra-datacenter transfers arefundamentally and conceptually different, as “70% of flowssend less than 10 KB and last less than 10 seconds” [15], andare moreover highly sensitive to delays. These characteristicsare in stark contrast to the bulk and replication-heavy transfersbetween datacenters, which allow for some delay and last forminutes to days, and hence intra-datacenter traffic managementrelies heavily on e.g. oblivious route management.

1) Handling uncertainty in traffic demand. The dynamics ofinteractive traffic and bulk transfer traffic across datacentersneed to be thoroughly considered to develop more efficientsolutions. Although high-priority interactive traffic can belargely predicted over short time frames [2], [4], mispredictionis inevitable and can lead to inaccurate leftover bandwidthestimates that may degrade the service quality of bulk trans-fers. Future research focuses on more accurate predictionsof interactive traffic that in turn can be leveraged for morereliable solutions for bulk transfers. To facilitate accurate traf-fic prediction and WAN optimization, designing effective andfine-grained network telemetry systems can be helpful to gainfurther insights into the current network state (e.g., detectingchanges in traffic patterns). In production datacenters, transferrequests typically arrive at the system in an online fashionand disclose their information after arrivals. Hence researchon bulk traffic prediction would also be beneficial to designingadvanced online solutions. Another open problem is to developinformation-agnostic approaches that can efficiently allocatetransfers without (nearly) complete knowledge of large transfertraffic or interactive traffic.

2) Study of various scheduling policies. Networking re-searchers have proposed many seminal but also new policiesfor scheduling intra-DC traffic. From a theoretical perspective,these policies can also be applicable to schedule inter-DC bulktransfers. However, it is not well understood if datacenterscheduling policies also work for such traffic well. So far,there is only one attempt ( [10]) that investigates severalscheduling policies and focuses on the average completiontime of P2MP transfers. A broader and better understandingon how existing datacenter scheduling policies impact bulktransfers with different communication patterns, performancemetrics, and objectives is an open problem. Compared todatacenter networks with a single bottleneck, an inter-DCnetwork may have multiple bottlenecks. Any scheduling policyfor bulk transfers should thus consider the more complexenvironments and unique transfer characteristics.

3) Optimizing the network for a mix of transfer patterns,constraints, and objectives. Datacenter networks typically hosta large collection of different cloud services that generate amixture of bulk transfers among datacenters with differentcommunication patterns, performance metrics and objectives.A provider may need to handle both P2P and P2MP transfers,

transfers desiring timely completion and and also a mixtureof objectives such as maximizing the number of deadline-meeting requests and minimizing transmission cost. Such avariety of data transfer workloads introduces more complexity,and efficiently optimizing them is an open problem.

4) Theoretical and mathematical analysis of inter-datacenter networks. Most current works on bulk transferoptimization problems rely on greedy and heuristic algorithms.At the same time, very few of them investigate algorithms withtheoretical guarantees and bounds. Theoretically analyzing thecomplexity and optimality scenarios of existing transfer algo-rithms is similarly not well-understood, except for simplifiednetwork models with single bottlenecks. However, multiplebottlenecks often occur in inter-DC WANs in practice. Anotheropen problem is to design new algorithms with theoreticalbounds for bulk transfer optimizations, especially in networkswith multiple bottlenecks.

5) Leveraging advanced hardware features to further op-timize inter-datacenter networks. Emerging technological in-novations in reconfigurable network technologies offer thenetworking community new dimensions to optimize the ex-plosively growing datacenter workloads. However, only fewapproaches utilize this technology to improve the performanceof inter-DC bulk transfers, e.g,. by reducing the completiontime of P2P transfers [12] and maximizing the number ofdeadline-meeting P2MP transfers [9]. Further exploiting thebenefits of reconfigurable network technology to improveperformance metrics and scheduling goals is an open issue.

6) Designing a common framework to capture and simplifyinter-datacenter traffic optimization. Datacenter workloads aregrowing explosively [6] and this trend will continue in next-generation networks, driven by increasing internet penetra-tion and emerging new applications such as Augmented andVirtual Reality and pervasive artificial intelligence. Next-generation networks will not only need to handle additionalacross-datacenter workloads, but also new complicated transferproblems with different communication patterns, performancemetrics, and objectives. The conventional practice to handlenew data transfer problems usually involves addressing newoptimizations, which requires significant manual effort andexpertise to express, non-trivial computation, and carefullycrafted heuristics. In the future, it is necessary and an openproblem to develop a general framework that can simplify themanual efforts in optimizing transfer allocation tasks.

7) Learning-based inter-datacenter traffic optimization.Given the rapid development of machine learning and itssuccessful application to many complex network problems,we believe machine learning can also be applied to addressthe increasingly complex bulk transfer problems. For example,tailored machine learning algorithms can intelligently choosethe right routing and scheduling policies and allocate cost-efficient transmission rate for mixed transfer workloads. How-ever, corresponding research has been sparse and bridging thisgap will be of high interest for future deployments. Moreover,to conduct such research, routing datasets from actual inter-DCnetworks need to be shared with researchers by operators.

7

V. CONCLUSIONS

The study and optimization of bulk data transfers overinter-DC WANs is a relatively new research area with manytechnical challenges that still need to be addressed. Inter-DCbulk transfers can lead to different and unique communicationpatterns, requiring various performance metrics and intro-ducing novel optimization objectives. We presented a shortreview on existing transfer techniques for developing efficienttransfer allocation solutions, focusing on transmission (e.g.,routing, rate allocation and delivery scheme), scheduling (e.g.,policies), and other techniques such as admission control andtopology control. We concluded by discussing several openresearch questions and future challenges that need furtherinvestigation from the networking research community.Acknowledgment. This work was partially supported by theNational Key Research and Development Program of Chinaunder Grant 2019YFB1802803; the PCL Future Greater-BayArea Network Facilities for Large-scale Experiments andApplications under Grant PCL2018KP001; the European Re-search Council (ERC) through the European Union’s Horizon2020 Research and Innovation Programme (AdjustNet: Self-Adjusting Networks) under Agreement 864228.

REFERENCES

[1] “Hyperscale Data Center Count Passed the 500 Milestonein Q3,” accessed on October 31, 2019. [Online]. Avail-able: https://www.srgresearch.com/articles/hyperscale-data-center-count-passed-500-milestone-q3

[2] C.-Y. Hong, S. Kandula, R. Mahajan et al., “Achieving High Utilizationwith Software-Driven WAN,” in ACM SIGCOMM Computer Communi-cation Review, vol. 43, no. 4. ACM, 2013, pp. 15–26.

[3] S. Jain, A. Kumar, S. Mandal et al., “B4: Experience with a GloballyDeployed Software Defined WAN,” in ACM SIGCOMM ComputerCommunication Review, vol. 43, no. 4. ACM, 2013, pp. 3–14.

[4] H. Zhang, K. Chen, W. Bai et al., “Guaranteeing Deadlines for Inter-Datacenter Transfers,” IEEE/ACM Transactions on Networking (TON),vol. 25, no. 1, pp. 579–595, 2017.

[5] S. Kandula, I. Menache, R. Schwartz, and S. R. Babbula, “Calendaringfor Wide Area Networks,” in ACM SIGCOMM computer communicationreview, vol. 44, no. 4. ACM, 2014, pp. 515–526.

[6] M. Jimenez and H. Kwok, “Building Express Backbone:Facebook’s new long-haul network,” accessed on October 29,2019. [Online]. Available: https://engineering.fb.com/networking-traffic/building-express-backbone-facebook-s-new-long-haul-network/

[7] N. Laoutaris, M. Sirivianos, X. Yang, and P. Rodriguez, “Inter-Datacenter Bulk Transfers with NetStitcher,” in ACM SIGCOMM Com-puter Communication Review, vol. 41, no. 4. ACM, 2011, pp. 74–85.

[8] L. Luo, Y. Kong, M. Noormohammadpour, Z. Ye, G. Sun, H. Yu, andB. Li, “Deadline-Aware Fast One-to-Many Bulk Transfers over Inter-Datacenter Networks,” IEEE Transactions on Cloud Computing, 2019.

[9] L. Luo, K.-T. Foerster, S. Schmid, and H. Yu, “DaRTree: Deadline-Aware Multicast Transfers in Reconfigurable Wide-Area Networks,” inIEEE/ACM IWQoS, 2019.

[10] M. Noormohammadpour, C. S. Raghavendra, S. Kandula, and S. Rao,“QuickCast: Fast and Efficient Inter-Datacenter Transfers using Forward-ing Tree Cohorts,” in IEEE INFOCOM, 2018, pp. 225–233.

[11] W. Li, X. Zhou, K. Li et al., “TrafficShaper: Shaping Inter-DatacenterTraffic to Reduce the Transmission Cost,” IEEE/ACM Transactions onNetworking, vol. 26, no. 3, pp. 1193–1206, 2018.

[12] X. Jin, Y. Li, D. Wei et al., “Optimizing Bulk Transfers with Software-Defined Optical WAN,” in SIGCOMM. ACM, 2016, pp. 87–100.

[13] Y. Zhang, J. Jiang, K. Xu, X. Nie, M. J. Reed, H. Wang, G. Yao,M. Zhang, and K. Chen, “BDS: A Centralized Near-Optimal OverlayNetwork for Inter-Datacenter Data Replication,” in EuroSys, 2018, pp.1–14.

[14] H. Yu, H. Qi, and K. Li, “WECAN: an Efficient West-East ControlAssociated Network for Large-Scale SDN Systems,” MONET, vol. 25,no. 1, pp. 114–124, 2020.

[15] A. Roy, H. Zeng, J. Bagga, G. Porter, and A. C. Snoeren, “Inside thesocial network’s (datacenter) network,” in Proceedings of the 2015 ACMConference on Special Interest Group on Data Communication, 2015,pp. 123–137.

BIOGRAPHIES

LONG LUO is currently working toward the PhD degree fromthe University of Electronic Science and Technology of China(UESTC). She received the BS degree in communicationengineering from Xi’an University of Technology in July2012 and MS degree in communication engineering fromthe UESTC in July 2015. Her research interests includesoftware-defined networks, inter-datacenter traffic engineeringand data-driven networking.

HONGFANG YU is a Professor at University of ElectronicScience and Technology of China. She received the BSdegree in electrical engineering in 1996 from XidianUniversity, and the MS and PhD degrees in communicationand information engineering in 1999 and 2006, respectively,from the University of Electronic Science and Technologyof China. From 2009 to 2010, she was a visiting scholarat the Department of Computer Science and Engineering,University at Buffalo (SUNY). Her research interests includeSDN/NFV, data center network, network for AI system andnetwork security.

KLAUS-TYCHO FOERSTER is a Postdoctoral Researcher atthe Faculty of Computer Science at the University of Vienna,Austria since 2018. He received his Diplomas in Mathematics(2007) & Computer Science (2011) from BraunschweigUniversity of Technology, Germany, and his PhD degree(2016) from ETH Zurich, Switzerland, advised by RogerWattenhofer. He spent autumn 2016 as a Visiting Researcherat Microsoft Research Redmond with Ratul Mahajan, joiningAalborg University, Denmark as a Postdoctoral Researcherwith Stefan Schmid in 2017. His research interests revolvearound algorithms and complexity in the areas of networkingand distributed computing.

MOHAMMAD NOORMOHAMMADPOUR received the Ph.D.degree from the Ming Hsieh Department of ElectricalEngineering, University of Southern California, USA. He iscurrently a Research Scientist with the Network Planningand Risk team at Facebook, Menlo Park, USA. His researchinterests include Network Risk Analysis and CapacityPlanning, as well as Distributed Systems Optimization andScaling in general.

STEFAN SCHMID is a Professor at the Faculty of ComputerScience at the University of Vienna, Austria. He receivedhis MSc (2004) and PhD degrees (2008) from ETH Zurich,Switzerland. In 2009, Stefan Schmid was a postdoc at TUMunich and the University of Paderborn, between 2009 and2015, a senior research scientist at the T-Labs in Berlin,Germany, and from the end of 2015 till early 2018, anAssociate Professor at Aalborg University, Denmark. Hisresearch interests revolve around fundamental and algorithmicproblems in networked and distributed systems.