NEW PARADIGMS FOR MANAGING THE COMPLEXITY ANDIMPROVING THE PERFORMANCE OF ENTERPRISE NETWORKS.

by

Theophilus A. Benson

A dissertation submitted in partial fulfillment ofthe requirements for the degree of

Doctor of Philosophy(Computer Sciences)

at the

UNIVERSITY OF WISCONSIN–MADISON

2012

Date of final oral examination: 06/29/2012

The dissertation is approved by the following members of the Final Oral Committee:Srinivasa A. Akella, Associate Professor, Computer ScienceSuman Banerjee, Associate Professor, Computer ScienceMichael Swift, Assistant Professor, Computer ScienceJennifer Rexford, Professor, Computer ScienceParameswana Ramanathan, Professor, Electrical and Computer Engineering

© Copyright by Theophilus A. Benson 2012All Rights Reserved

i

To the many that have influenced my life, namely my family, my friends, and my foes(time and sleep).

ii

acknowledgments

"Since men almost always tread the paths made by others and proceed in theiraffairs by imitation, although they are not completely able to stay on the path ofothers nor attain the skill of those they imitate, a prudent man should alwaysenter those paths taken by great men and imitate those who have been mostexcellent, so that if one’s own skill does not match theirs, at least it will havethe smell of it; and he should proceed like those prudent archers who, aware ofthe strength of their bow when the target they are aiming at seems too distant,set their sights much higher than the designated target, not in order to reach tosuch a height with their arrow but rather to be able, with the aid of such a highaim, to strike the target"

–The Prince, Niccolo Machiavelli.

To my advisor and mentors, all great men whom only by emulating have Iachieved the majority of my deeds.

First and foremost, I would like to thank my advisor who took on thechallenge of guiding me down this path and who, through the years, hasremained supportive of me and my ideas. Under his tutelage, I have growntremendously both intellectually and personally. My aim is to one day replicatehis ability to assimilate new information and develop ideas that leave a roomfull of experts amazed.

I would also like to thank my mentor, David A. Maltz, for his guidanceand support over the last five years. Despite trying for five years, I have yetto accurately replicate his eloquence and foresight. I would also like to thankAnees Shaikh who bought into my vision of cloud computing and devotedprecious resources to nourish my ideas. I hope to one day gain his ability tolook beyond the noise and choose truly important and impactful problems.

I would like to thank members of my thesis committee, Jennifer Rexford,Michael Swift, Suman Bannerjee and Parmesh Ramanathan, for their invaluablecomments in shaping up my thesis. I would also like to thank my collaborators,Aman Shaikh and Ming Zhang, for their comments and feedback during the

iii

writing of this thesis.To my friends, all great individuals whose brilliance, support, and persever-

ance have spurred within me what I required to follow the path.I would especially like to thank Richard Guerra, Gabriela Jacques Da Silva,

and Vera Tatel, my dearest friends who willingly offered daily words of supportduring this journey. I would also like to thank my good friends whose impacton my life cannot be expressed within a few words: Anna Kaltenboek, RyanVinelli, Marc Weintraub, Olusheun Olupitan, Elise Wu, Gary Hsieh, MauricoChan, Ramya Raghavendra, Jamil Tomlin, Saira Sibazi, and Mirrette Kouchouk.

Having found myself displaced in the Midwest, several individuals endeav-ored to help me transform this landscape into home. Among them were: MikeBlodgett, Dave Plonka, Archit Gupta, Maneesh Minsha, Aaronsan Chew, Eun-soo Seo, and Tristen Mapp. Finally, I would like to thank the brilliant studentsat Wisconsin that have taken on the role of being my sounding board; MattRenzelmann, Asim Kadav, Aaron Gember, Ashok Anand, and Sayandeep Sen.

To my family, the best of supporters, whose vocal and tacit guidance haveset me on this path and continue, to this day, to impact the way I thread thispath.

I would like to thank my mother who has given up an immeasurable amountto raise me. I can only hope that my meager accomplishments will one daymake up for the dreams she has sacrified for me. I would also like to thank mybrothers, Fred and Larry, both of whom have played a huge role in shaping theway I view and interact with the world.

iv

contents

Contents iv

List of Tables vii

List of Figures ix

1 Introduction 11.1 Enterprise Networks 31.2 The Challenges in Managing the Reliability and Performance of En-

terprise Networks 81.3 Current Approaches to Managing Enterprise Networks 111.4 New Approaches to Managing Enterprise networks 131.5 How to Read this Thesis 17

2 Measuring the Complexity of Enterprise Configuration Management 192.1 Data Set: Studied Networks 192.2 Overview of Network Design and Configuration 212.3 Implementation Complexity: Reference Chains 242.4 Implementation Complexity: Router Roles 302.5 Inherent Complexity 342.6 Complexity Model: Reachability Sets 352.7 Complexity Metrics 382.8 Evaluating Inherent Complexity: Insights from Operator Interviews 402.9 Summary of Complexity Metrics 43

3 Policy Units 453.1 Methods for Discovering Policy Units 463.2 Empirical Study of Policy Units in Enterprise Networks 543.3 Summary of Policy Units 58

4 Demystifying Configuration Challenges and Trade-Offs in ISP Ser-vices 60

v

4.1 Background 624.2 Approach 684.3 Configuring Services 704.4 Customer Provisioning Under the Microscope 814.5 Trade-offs in Configuration Complexity 854.6 Conclusions 90

5 Understanding Data Center Traffic Characteristics 925.1 Datasets and Overview of Data Centers 955.2 Applications in Data Centers 1005.3 Application Communication Patterns 1035.4 Network Communication Patterns 1105.5 Implications for Data Center Design 1205.6 Summary 124

6 MicroTE: Fine Grained Traffic Engineering for Data Centers 1266.1 Background 1286.2 Comparative Study 1306.3 Design Requirements for an Ideal Traffic Engineering Technique 1336.4 MicroTE: Architecture 1376.5 Implementation 1456.6 Evaluation 1456.7 Conclusion 154

7 Related Work 1557.1 Enterprise Configuration Management 1557.2 Data Center Networks 158

8 Conclusion and Future Work 1618.1 Contributions 1618.2 Impact 1638.3 Lessons learned 1638.4 Emerging Trends 1658.5 Directions for Future Research 166

vi

References 170

vii

list of tables

2.1 Studied networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.2 Complexity due to referential dependence. Networks where we

validated results are marked with a “Y.” . . . . . . . . . . . . . . . 272.3 Roles extracted from ACLs. . . . . . . . . . . . . . . . . . . . . . . 332.4 Inherent complexity measures. . . . . . . . . . . . . . . . . . . . . 43

3.1 Pseudocode for extracting policy units from subnet reachabilitymatrix, assuming each SRS is represented using an ACL. . . . . . 53

3.2 Properties of the 5 networks studied, including the number of policyunits found in each. . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.1 Configuration for a VPN PE. . . . . . . . . . . . . . . . . . . . . . . 664.2 BGP configuration for a route-reflector. . . . . . . . . . . . . . . . 684.3 Percentage of ISP devices used by the five services studied. The

devices used by a service include both the PEs and the RRs, butnot the MPLS core devices since the core devices are shared by allservices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4.4 Number of routes and difficulty for maintaining each of the differentVPN designs. Here, C is the number of customer routes, E is thenumber of customer sites, and H is the number of hubs. . . . . . . 89

4.5 Number of peering sessions, routes and difficulty for maintainingeach of the different BGP control-plane designs. Here, N is thenumber of PEs, R is the number of route-reflectors,M is the numberof different planes, and T the total number of routes contributed byall customers. We also assume that each PE is connected to two RRsfor robustness purposes. . . . . . . . . . . . . . . . . . . . . . . . . 91

5.1 Summary of the 10 data centers studied, including devices, types ofinformation collected, and the number of servers. . . . . . . . . . 96

5.2 The number of packet trace collection locations for the data centersin which we were able to install packet sniffers. . . . . . . . . . . . 98

viii

5.3 The distribution for the parameters of each of the arrival processesof the various switches. . . . . . . . . . . . . . . . . . . . . . . . . 109

5.4 The distribution for the parameters of each of the arrival processesof the dominant applications on each switch. . . . . . . . . . . . . 109

6.1 Runtime (in seconds) of algorithm for different DC sizes and fordifferent fractions of ToR pairs with predictable traffic. . . . . . . 153

ix

list of figures

1.1 A typical enterprise network spans 3 components: (A) the campusnetworks, (B) the ISP services, (C) the data center networks. . . . 3

1.2 Using ISP services, an enterprise can transform its network into aglobal intranet where the data centers and campuses are intercon-nected with point-to-point links. . . . . . . . . . . . . . . . . . . . 3

1.3 The campus network is extremely diverse, containing devices fromdifferent vendors and fulfilling requirements for different sets ofusers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1 (a) Distribution of configuration file size across networks. (b) Frac-tion of configuration dedicated to configuring each aspect of routerfunctionality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.2 A sample configuration file. . . . . . . . . . . . . . . . . . . . . . . 222.3 Example of a configuration template. . . . . . . . . . . . . . . . . . 302.4 A toy network with 8 subnets and 5 routers. The different constituent

sets that play a role in the reachability set for the A→C path areshown. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

2.5 Computing the first- and second-order metrics for inherent com-plexity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.6 This figure shows the clusters of routers in Univ-2 that have similarreachability to the given destination router. The X axis is the sourcerouter ID. The Y axis is the distance between the centers of theclusters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.1 An enterprise with 3 departments and 4 policy units. The Prod-ucts department consists of two units, one of which corresponds toadministrators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

x

3.2 (a) An SRS matrix for a network with three subnets. For this example,the constraints placed by the network apply only to the source anddestination IPs. (b) The rectangles and policy subunits after thesubnet-level reachability sets have been decomposed. (c) The finalpolicy units. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.3 Structure of policies in the networks studied. Figure (a) is a CDF ofpolicy units as a fraction of the network devices spanned. Figure(b) shows the cumulative fraction of end-points in policy units (as aCDF). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

3.4 Policy in Univ-1 is "on by default," with 70% of hosts able to reachnearly all subnets and 30% able to reach all subnets. . . . . . . . . 57

3.5 Policy in Univ-3 places most hosts into "default on" in units 7-15,and a few hosts into "default off" units. . . . . . . . . . . . . . . . . 59

4.1 Architecture for a VPN service in an ISP. . . . . . . . . . . . . . . 634.2 Longitudinal analysis of the referential graphs in service PE devices. 714.3 Longitudinal analysis of the types of stanzas in the referential graph

(averaged over PEs). . . . . . . . . . . . . . . . . . . . . . . . . . . . 734.4 Longitudinal analysis of configuration templates. . . . . . . . . . 754.5 Longitudinal analysis of changes in templates. . . . . . . . . . . . 764.6 Configuring core routers to integrate services. . . . . . . . . . . . 784.7 Properties of the per-service RR referential graph. . . . . . . . . . 794.8 Longitudinal analysis of the types of stanzas in the referential graph

(Averaged over RRs). . . . . . . . . . . . . . . . . . . . . . . . . . . 794.9 Per customer properties for all customers. . . . . . . . . . . . . . . 834.10 Referential graph for provisioning VPLS customers. . . . . . . . . 844.11 Examining reuse within a network. . . . . . . . . . . . . . . . . . . 864.12 Differences between vendor configuration. . . . . . . . . . . . . . 87

5.1 Canonical 3-Tier data center topology. . . . . . . . . . . . . . . . . 99

xi

5.2 Classification of network traffic to application using Bro-Id. Each ofthe sniffers sees a very different mix of applications, even thoughthe first 4 sniffers are located on different switches in the same datacenter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

5.3 CDF of the distribution of the number of flows at the edge switch (a)and the arrival rate for flows (b) in EDU1, EDU2, EDU3, and PRV2. 105

5.4 CDF of the distribution of the flow sizes (a) and of flow lengths (b)in PRV2, EDU1, EDU2, and EDU3. . . . . . . . . . . . . . . . . . . 106

5.5 ON/OFF characteristics: Time series of Data Center traffic (numberof packets per time) binned by two different time scales. . . . . . 107

5.6 CDF of the distribution of the arrival times of packets at three of theswitches in PRV2. The figure contains best fit curve for lognormal,Weibull, Pareto, and Exponential distributions, as well as the leastmean errors for each. . . . . . . . . . . . . . . . . . . . . . . . . . . 108

5.7 The ratio of Extra-Rack to Intra-Rack traffic in the data centers. . 1115.8 CDF of link utilizations (percentage) in each layer. . . . . . . . . . 1135.9 A CDF of the fraction of times that links in the various layers are

hot-spots. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1155.10 Time series of the fraction of links that are hot-spots in the core layer

for CLD1, CLD2, and CLD5. . . . . . . . . . . . . . . . . . . . . . . 1165.11 A CDF of the number of bits lost across the various layers. . . . . 1175.12 A CDF of the utilization of links with discards. . . . . . . . . . . . 1185.13 Time-of-Day/Day-of-Week traffic patterns. . . . . . . . . . . . . . 1195.14 Difference between the peak and trough utilization. . . . . . . . . 1195.15 The first bar is the ratio of aggregate server traffic over Bisection BW

and the second bar is the ratio of aggregate server traffic over fullbisection capacity. The y-axis displays utilization as a percentage. 122

6.1 Distribution of the MLU for optimal, ECMP and Spanning Tree on acanonical tree topology. MLU greater than 1 indicates loss. . . . 131

6.2 (a) Distribution of the fraction of total traffic demand and (b) Distri-bution of the mean run-length for top 100 ToR pairs. . . . . . . . . 135

6.3 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

xii

6.4 Flow chart of the logic within the routing component. . . . . . . . 1446.5 Comparison of MicroTE, Optimal and ECMP for (a) UNV and (b)

CLD. For CLD, Micro and ECMP curves overlap. . . . . . . . . . . 1466.6 Comparison of MicroTE, Optimal and ECMP for UNV; ideal case

with application level comparison. . . . . . . . . . . . . . . . . . . 1476.7 (a) Low and Medium predictability (between 0% and 75% of the

matrix is predictable), and (b) High predictability (greater than 75%of the matrix is predictable). MLU greater than 1 indicates loss. . 149

6.8 Comparison of different threshold values for determining predictabil-ity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

NEW PARADIGMS FOR MANAGING THE COMPLEXITY ANDIMPROVING THE PERFORMANCE OF ENTERPRISE NETWORKS.

Theophilus A. Benson

Under the supervision of Professor Aditya AkellaAt the University of Wisconsin-Madison

Networks are a crucial and integral aspect of modern enterprises, with the pro-ductivity and competiveness of many enterprises significantly intertwined withthe performance and reliability of their networks. Yet despite their importance,these networks remain highly susceptible to poor performance, failures, andoutages.

We posit that the fragility of enterprise networks is largely a byproductof the complexity involved in managing conflicting global policies and thatpoor performance, especially within data centers, is largely a byproduct ofinsufficient knowledge about the characteristics of the traffic within these datacenters.

We argue that a key way to harden these networks against failures andfragility is to understand complexity. To this end, we have developed a set ofsimple yet highly effective models of complexity within enterprise networksand used these models as first-order principles in designing practical frame-works that improve the performance and reliability of these networks. Theseframeworks mitigate complexity by enabling large enterprises to debug theirnetwork’s configuration and by informing the design of more efficient andreliable services. Our evaluation of these frameworks illustrates how reducingcomplexity can lead to manageable networks, thus reducing misconfigurationsand promoting innovative changes. More broadly, we believe that understand-ing complexity has several benefits: in the short run, knowledge of complexityleads to the development of mechanisms that foster better performance andhigher resiliency; and in the long run, the dialogue created by understandingcomplexity will shape the design of future networking interfaces and abstrac-tions.

xiii

In terms of performance, we focus on understanding a specific segment ofenterprise networks: data centers. To better understand data centers, we con-ducted an empirical study of the network traffic in 10 data centers and showedhow observations from this study can be used to design more efficient trafficengineering techniques for current data centers. Using these observations, wedeveloped MicroTE, a system that adapts to traffic variations by leveraging theshort-term and partial predictability of the traffic matrix. Through evaluations,we show that MicroTE performs close to the optimal solution and imposesminimal overhead on the network, making it apt for current and future datacenters.

Aditya Akella

1

1 introduction

Over the last few years, networks have become an invaluable asset to enterprises,with the productivity and revenues of these enterprises intrinsically tied totheir underlying networks. Enterprises use these networks either as a venuefor offering services to customers or as crucial resources with which to performcore business activities. Sadly, despite the importance of enterprise networks,they remain plagued by performance, availability, and reliability issues. Infact, most recent articles about the operations of these networks highlight theseissues.

Below are a few recently released articles:

• A survey released by Yankee Group [47] in 2005 attributed over 60% ofthe network outages within enterprise campuses to operators misconfig-uring network devices. This fact indicates that productivity can be easilyimproved by reducing operator misconfiguration.

• In July and August 2011, a network misconfiguration in a Tier 3 ISP,Fusion [70, 71], took down connectivity to customers across the city ofSan Francisco. Disruptions such as these within ISPs indirectly affectenterprises by limiting the ability of employees to work from home, bypreventing customers from using and purchasing enterprise offerings,and by isolating different portions of an enterprise’s intranet.

• In 2008, Amazon reported that “every 100ms of latency cost them 1% insales” [38]. This translates to a loss of roughly $300 million in revenue. 1

Given the increase in Amazon’s revenue we expect the monetary loss tobe larger now. To battle this loss of competitiveness, enterprises employ avariety of expensive and often ineffective tactics such as over provisioningand deeper buffers.

• A survey by TABB group [66] reported that for financial institutions, a5 millisecond latency could result in a loss of $4 million in revenues. In

1Revenue numbers calculated based on quarterly reports.

2

addition to over provisioning their data centers, these enterprises alsoinvested as much as $15 billion in 2009 [56] to physically collocate theirdata centers to reduce the impact of an ISP’s services on their networks.

These articles illustrate a simple fact: the performance, availability, andreliability of ISP networks (more notably the services connecting enterpriseresources), data center networks (used to host large collections of enterpriseresources), and campus networks (enterprise buildings) affect the productivityand revenues of an enterprise. As a means of protecting revenue and pro-ductivity, enterprises employ legions of IT operators with the sole objectiveof managing the availability, reliability, and performance of these networks.The biggest roadblock to achieving this objective, be it preventing these out-ages from occurring or modifying the network to improve its performance, isthat the mechanisms, tools, frameworks, and processes required for effectivelymanaging enterprise networks are missing. Motivated by this observation, thisthesis focuses on answering two fundamental questions:

• What sort of insights are required to develop a rich set of network modelsthat capture: (1) aspects of network configurations that impact availabilityand reliability; and (2) aspects of network traffic that impact performance?

• Can we apply the models in the form of mechanisms, systems, or frame-works to aid operators in managing enterprise networks? If so, whatchallenges must we overcome, in terms of scalability, efficiency, and rep-resentation, in developing these mechanisms, systems, and frameworks?

The remainder of this chapter describes enterprise networks in more detail,with Section 1.1 describing the components that make up an enterprise net-work, Section 1.2 presenting the challenges that make them highly unreliableand unpredictable, Section 1.3 discussing current solutions and their draw-backs, Section 1.4 describing the contributions of this thesis, and finally withSection 1.5 providing a road map for the rest of the thesis.

3

Figure 1.1: A typical enterprise network spans 3 components: (A) the campusnetworks, (B) the ISP services, (C) the data center networks.

Figure 1.2: Using ISP services, an enterprise can transform its network into aglobal intranet where the data centers and campuses are interconnected withpoint-to-point links.

1.1 enterprise networks

The performance and availability of enterprise networks are affected by manydifferent networked components. We believe that only by viewing the enterprisenetwork as a collection of these components and by developing holistic solutionsthat address the unique challenges in each of these components can we produce

4

Figure 1.3: The campus network is extremely diverse, containing devices fromdifferent vendors and fulfilling requirements for different sets of users.

solutions that impact the end-to-end avialability and performance of enterprisenetworks . In Figure 1.1, we present the different components and describehow they work together to provide the abstraction of an enterprise network.Starting at the edges are the enterprise campus networks, labeled A, whichhost the enterprise’s employees and the data center networks, labeled C, whichhost server and storage resources. Data centers are large dedicated clustersmaintained by a single entity. This definition encompasses data centers usedinternally by corporations, data centers used for hosting services for externalusers, and data centers used to host a cloud. Following from this, a data centernetwork is then the network within the data center interconnecting the serversand storage resources. In the middle, labeled B, are the ISP services used tointerconnect different enterprise sites (campuses and data centers). Unlike thecampus and data center networks, enterprises do not control the ISP networks;however, the performance and availability of the networks they do control,namely the campuses and data centers, often are tied to the performance andavailability of the service provided by an ISP. Next, we briefly expand on theunique properties of each of these three components.

5

Campus Networks

The campus network is the Local Area Network (LAN) within a building thatconnects the employees to each other and to the global Internet. For example, alarge corporation such as Bank of America (BoA), with branches in differentlocations, would have several campus networks, one at each location. The NewYork branch would have a campus network, consisting of switches, routers, andmiddle-boxes, connecting the laptops, desktops, mobile phones, and tabletsused by the employees in the building. The Boston branch would have itsown distinct campus network for interconnecting the employees housed in theBoston building.

Several of the more interesting properties of campus networks arise out ofthe fact that campus networks connect all employees within the building.

First, different employees have different requirements; thus network oper-ators design and configure campus networks to support a large array of globalnetwork reachability policies and objectives, including but not limited to, access con-trol and quality of service. Furthermore, given that different devices supportdifferent sets of functionality, operators employ a large array of heterogeneousdevices to enforce and support the different user requirements. For exam-ple, BoA’s Boston branch may consist of several departments: finance, humanresources (HR), and payroll. HR uses a 3-tier web service for their daily activi-ties. The finance department uses several web services. Finally, payroll usesa line-of-business application. All employees of the bank are able to accessthe finance department’s web services. However, due to privacy reasons onlymembers of the payroll and HR department can access the payroll department’sline-of-business application and HR’s web services, respectively.

Second, enterprises are organic entities with a constantly changing employeebase and a constantly evolving set of global requirements and responsibilitiesbeing assigned to each employee; thus, the campus network is an organic entity,constantly being modified to reflect changes in the global requirements and inemployee responsibilities. In the face of such changes, network operations mustensure that the network always enforces the enterprise’s global reachabilitypolicies and objectives. Failure to do this can have disastrous consequences on

6

the network. Returning to the earlier example, over time BoA may decide thataccess to the finance web services should be limited to employees within thefinance department and certain HR application servers should be accessibleto all BoA employees. This change necessitates careful modification of thenetwork; failure to do this can result in outages to certain servers or perhapssecurity holes that render sensitive data accessible to attackers!

ISP Services

Interconnecting the different campus and data center networks are the servicesoffered by an ISP. Many of these services provide enterprises with the illusionthat they are connected by a single link rather than by the collection of routersand links that make up the Internet. The most notable examples of theseservices include VLPS [49] (Virtual Private LAN Service) and VPN [50] (VirtualPrivate Network), both of which provide a point-to-point or point-to-multi-point abstraction between campuses and data centers.

To interconnect the Boston branch, New York branch, and the Washing-ton data center, Bank of America can purchase a VPN service from AT&T orsubscribe to traditional best-effort connectivity. The advantages of VPN, andother such enterprise services, over traditional best-effort are three-fold: first,security, the packets traversing the WAN are automatically encrypted; second,QoS guarantees, ISPs often provide enterprises with the option to obtain QoSas part of the service; and third, the point-to-point link abstraction, this abstrac-tion of direct connectivity between geographically distributed sites allows anenterprise to form its own private intranet across all sites. Bank of America’sintranet would span the sites in Boston, New York, and Washington. With VPN,BoA’s intranet would look like Figure 1.2 instead of Figure 1.1.

Managing these ISP services revolves around setting up and modifyingthe configuration for the specific services used by different enterprises. Aninteresting property of this network arises because enterprises do not directlycontrol its configuration or management; however, the network’s configuration androuting design can adversely impact the enterprise. Thus ISPs strive to configurethese services to meet the demands of their enterprise customers. Moreover,

7

ISP operators constantly modify these services in accordance with changes inthe requirements of enterprise customers. For example, should BoA decide toadd a fourth site or to increase bandwidth between sites, AT&T would needto conduct the laborious task of modifying configuration of BoA’s instance toaccommodate this request.

Data Center Networks

Unlike with the campus networks, operators manage data center networks tosupport a large concerntration of storage and processing resources. Enterprisesuse these data centers to perform batch (non-interactive) operations, as well asto perform computationally or storage intensive real time processing such asanalytics. Returning to Bank of America, in addition to campuses in New Yorkand Boston, BoA owns a data center in Washington. The finance departmentuses this data center for intensive data processing and analytics.

Three interesting properties arise from supporting these batch and computa-tionally intensive applications. First, data centers are homogeneous, consistinglargely of identical network and servers; the ability to run tasks at any locationin the data center demands uniformity across all locations within the data cen-ters. Second, within data centers, operators place more emphasis on performance;servers generate a significant fraction of the network traffic and the effectivenessof these servers is a function of the performance of the network. Finally, many ofthese enterprises employ virtualization to simplify and speed up provisioningand deprovisioning of resources for these jobs. The ability of BoA’s financedepartment to generate revenue with its analytics applications can be hinderedby inefficiencies within the data center. For example, inefficiencies such asexcessive buffering or congestion will introduce latency problems resultingin poor performance for BoA’s analytics software and loss of revenue for thefinance department.

8

1.2 the challenges in managing the reliability and performance ofenterprise networks

The availability, reliability, and performance of an enterprise network, suchas Bank of America’s network, is affected by the availability, reliability, andperformance of any of the three components described above; the campus,the data center, or the ISP. The challenges involved in maintaining availability,reliability, and performance differ across the different components, and thusnecessitates different types of solutions.

Campus Network

The challenge of maintaining and enforcing global reachability objectives andpolicies within campus networks is inextricably linked to the fact that diversepolicies and goals are being configured across a network of diverse devices.

Challenge: Configuring with low-level commands. Campuses contain de-vices from multiple vendors often operating at different layers and providing adisparate range of network functionality. Operators embed global policies intothe network by configuring commands within different subsets of the network.The command set for these devices differs across vendors and, even within avendor, the commands differ across device types. Furthermore, each commandset contains thousands of commands, and each command contains a large num-ber of parameters and options. Ensuring availability is challenging in suchan environment, as it is difficult to reason about changes within a particularnetwork. For example, implementing the new global policy within BoA’s Bostoncampus, illustrated in Figure 1.3, that only finance employees have access tofinance equipment may require modification of several devices; however, thenetwork consists of a mixture of switches, firewalls, and routers from Ciscoand Juniper. An operator will need to re-implement similar policies withinJunos (Juniper’s language), IOS (Cisco’s language), and perhaps Force10’s con-figuration languages. As a result of this, operators can easily misconfigure thenetwork and thus create a network outage.

Challenge: Configuring in a distributed environment. In addition to

9

working with low level commands, operators must make changes consistentlyacross the network. Essentially the network is a distributed system consistingof the set of devices required to provide connectivity between the differentend points within the network. To implement the above example of enforcingsecurity rules, an operator must make changes consistently across the entirenetwork. Failure to do this will result either in a security violation (allowingpotential intruders into the network) or in black holes (preventing legitimatetraffic from traversing the network). For example, configuring all devices exceptrouter ”A” can result in a security violation permitting unauthorized access tofinance servers "B" and "C".

Challenge: Configuring in the dark. To accurately change a network, theoperator must first understand the goals and policies currently instituted withinthe network. However, these networks are poorly documented and thus op-erators must examine the configuration of the network to understand the im-plemented goals and policies. More specifically, network operators must scanthrough the low-level commands across a subset of the devices in the networkwhile creating a mental model of how the different pieces fit together: a taskthat is highly prone to human error and misconfiguration.

Implications: The implications of these challenges are: (1) making a changeis an extremely time-consuming and error-prone task; and (2) it is non-trivialto determine if a network enforces the desired network goals or to understandwhich policies are enforced in the face of unexpected events such as devicefailures.

ISP Network Services

In general, ISP network operators face similar configuration challenges as cam-pus network operators. And thus, the tools developed for enterprise campuseswill suffice for ensuring the availability and reliability of these networks. How-ever, most of the configurations within the ISP do not impact the performance oravailability of dependent enterprises. To this end, we focus on the configurationof the ISP services used by an enterprise and examine the choices operatorsmake in deciding how to configure such a services to provide acceptable per-

10

formance and availability to the enterprise customer. These choices range fromoptimizations for state considerations and optimizations for latency to selectionof well understood design choices or protocols

Challenge: Optimizing in the dark. The challenge then lies in understand-ing the set of trade-offs that network operators face when choosing optimiza-tions and designs with which to maintain and configure an enterprise service.More specifically, systematically capturing the impact of these trade-offs on thereliability and availability of the dependent enterprise is challenging.

Implications: The implications of this challenge are that ISP operators areforced: (1) to empirically determine, over time and through trial and error,the trade-offs between different service configurations and optimizations; and(2) to limit themselves to the set of empirically understood optimizations anddesign patterns, a small subset of applicable configuration and optimizations,effectively eliminating more efficient solutions.

Data Center Network

Currently, enterprises blindly apply techniques used with campus and ISPnetworks to data center networks. While data centers share many of the chal-lenges experienced within the campus networks, performance and policiesrelated to performance are of the utmost importance. Unfortunately, data centernetworks are not campus networks, and the performance techniques honed forcampuses fail to satisfy the performance requirements of data centers. In datacenters, unique applications, such as MapReduce in which servers exchangedata with each other, generate a significant fraction of the traffic on the network.These unique application patterns produce radically different communicationspatterns than those observed within campus and ISP networks.

Challenge: Designing in the Dark. The main challenge in addressing thesecommunication patterns arises because we, as a community, understand littleabout the nature and the properties of traffic within data centers. Withoutsuch knowledge, it is difficult to develop or determine the set of models andperformance-optimization techniques that can be used efficiently within datacenters. Even worse, without a thorough understanding of data center networks,

11

it is hard to know if there are other more efficient and effective techniques forsatisfying other global policies such as security, reliability and availability.

Implication: The implications of this challenge are that operators are forcedto: (1) blindly apply, within the context of data center networks, utility functionsused to evaluate the effectiveness and the efficiency of network protocols withinISP and campus networks; (2) blindly borrow techniques from other domainsto use within data center networks; and (3) evaluate novel network protocolsand architectures using unrealistic models.

1.3 current approaches to managing enterprise networks

The state-of-art and recently proposed approaches to managing these networksare limited in several ways. Next we describe the different proposals and discusstheir limitations.

Campus Network

Approaches to enforcing and maintaining global reachability policies rangefrom being proactive and eliminating misconfiguration, to being reactive anddetecting misconfiguration, to being centralized and eliminating the distributedand low-level configuration interface. The proactive approaches [21, 33] stat-ically inspect the network’s configuration and subject it to a set of rules andconstraints that detect misconfiguration. The strength of these techniques liesin the fact that they detect misconfigurations prior to deployment and thusprevent outages. However, these approaches can only detect the set of miscon-figurations that conform to the predefined set of rules and constraints. Once anoutage has occurred, reactive techniques [8] expedite the process of perform-ing problem diagnosis and root-cause analysis, thus reducing the duration ofoutages. Many of these reactive techniques apply machine learning techniquesto detect misconfiguration. Although reactive techniques speed up the processof resolving an outage, they require that the outage first occur. Redesigningthe configuration interface is, in spirit, a step towards the eliminating miscon-figuration and network outages. However, current attempts to redesign the

12

interface [9, 20, 22, 36] are ineffective because they either provide centralizationwithout fully eliminating low-level commands or propose a new configurationcommand set without first developing and understanding the right abstractions.

At the heart of the problem, these approaches fail to understand, analyze,and quantify complexity within the configuration of the networks. Anecdotalevidence suggests that the complexity arising from configuring in a distributedand low-level environment causes outages and misconfiguration. We arguethat understanding this complexity is the first step towards improving reliability, andreducing complexity can reduce the operational cost of enforcing global reachabilitypolicies. Furthermore, a thorough understanding of complexity can complementexisting approaches by reducing the number of rules and constraints to beproactively used on a network’s configuration; by speeding up the processof reactively scanning for a misconfiguration; and by providing insights thatcan guide the design of future control planes. While others [34, 51, 54] havesought to understand complexity, they limit themselves to specific commands,protocols, or functionality, thus limiting their generality across different typesof networks.

ISP Network

Much work has gone into hardening the configuration of ISP networks, improv-ing their predictability [72], reliability, and availability [31, 52]. Even withinthe context of ISP services, work has been performed to create tools for under-standing various ways of implementing these services so as to improve theperformance for the enterprise [42, 64], to simplifly troubleshooting [53, 75],or to improve utilization for the underlying ISP [48, 65]. However, the ques-tion of evaluating the management overhead associated with each of theseand other optimizations remains unanswered. More concretely, an operatorcannot systematically quantify the difficulty of evolving a service to supportthe constantly changing enterprise requirements given a set of optimizationsand design choices. More generally, the question of examining the implication ofconfiguration complexity on common operations decisions remains to be tackled. Thishas remained largely ignored because of our inability, as a community, to quan-

13

tify the impact of configuration on other aspects of network management andto determine which of the many daily decisions are truly non-trivial and thusshould be highly investigated and evaluated.

Data Center Network

Initial attempts to improve performance within data centers borrowed tried andtested techniques from traditional enterprise campus and ISP networks: ECMPand Spanning Tree. However, anecdotal evidence suggests that these techniquesperform inefficiently when applied to data center networks. Towards this endmany solutions [4, 5, 29, 35, 39, 46, 60, 76] have been developed to fill this void;however, many of these are designed and evaluated without knowledge ofthe properties of data centers or with little knowledge of the corpus of datacenters running in the wild, limiting their generality to a subset of the classesof operational data centers. Fundamentally, these approaches are limited bythe scarcity of models, workloads and simulators characterizing, analyzing,describing, and replicating the properties of the large array of operational datacenters. We argue that without the knowledge of realistic patterns and properties oftraffic within data centers, it is near impossible to generate and develop techniques thatwork effectively and efficiently under varying types of operational data centers.

1.4 new approaches to managing enterprise networks

In developing new approaches for managing the reliability, performance, andavailability of an enterprise network, we start by creating two sets of modelsand then develop a set of frameworks that use these models to improve thereliability, performance, and availability of one or more of the components of anenterprise network. Our models and frameworks aim to address the limitationsand to fix the implications of the current management approaches broached inthe previous two sections.

Starting with the campus networks, we focus on configuration management.We developed a set of models to quantify the difficulty operators face whenmodifying the network’s configuration to enforce and maintain global policies.

14

Given that the same configuration command sets are used within ISPs, wealso applied these models to understand the management overhead associatedwith maintaining the configuration of enterprise services within ISP networks.Around this model, we developed two frameworks. The first, called PolicyUnits, automatically mines the global policies actually being implementedwithin a network’s configuration. This framework uses our model to improvemanagement by allowing operators to easily verify the set of global policiesbeing enforced by the networks and to proactively determine the impact oftheir choices on the network. The second, a strawman approach, shows howour model can be integrated into an ISP’s daily operations and used to helpinform non-trivial design trade-offs such as control-plane design and vendorselection.

Switching to the data center, with an eye towards performance, we began byconducting an extensive study of traffic within different types of data centers.Through this study we developed an understanding of the characteristics oftraffic within data centers and used this understanding to: (1) develop a setof models that characterize the properties of traffic within a large range ofdata centers; (2) analyze the effectiveness of various architectural decisionswithin data center networks; and (3) determine the design requirements foran ideal traffic engineering technique for improving performance of a datacenter network. To illustrate that a practical system can be built atop ourdesign requirements, we designed and implemented MicroTE, a scalable andefficient traffic engineering technique that implements to the predeterminedrequirements.

Measuring the Complexity of Enterprise Configuration Management:Through operator interviews and anecdotal evidence, we determined that

an operator’s ability to manage the availability and performance of a networkdecreases as the network becomes more complex. Although complexity iscrucial to management, there is currently no way to systematically quantifyhow complexity may affect network management activities.

To this end, we developed a suite of complexity models [16] that describethe routing design and configuration of a network in a succinct fashion, whileabstracting away details of the underlying configuration languages.

15

To demonstrate the effectiveness of our metrics, we tested them on sevennetworks, including four university networks and three enterprise networks.Furthermore, we validated the results of our tests through interviews with theoperators of five of the networks. We showed that our metrics are an accuratepredictor of the difficulty faced by operators in reconfiguring their networks.As an accurate predictor of complexity, our metrics can simplify network designand management by facilitating comparison between alternative designs for anetwork.

Policy Units in Enterprise Networks: We now shift our focus to under-standing global reachability policies. These policies are implemented indirectlyusing network protocols and hence lie buried in device configuration files, orworse still, in the minds of the network operators. We aim to provide operatorswith a better understanding of these reachability policies

To this end, we designed a framework for capturing policy units, which arean abstract representation of how the policies implemented in a network applyto different network hosts [15]. The framework consists of a set of algorithmsthat parse the configuration files, simulate control plane protocols, discoverpaths between network hosts, determine data-plane restrictions on such paths,and group together network hosts with identical control and data restrictions.The grouping algorithm clusters endpoints based on how the control and datarestrictions treat communication between the endpoint and the rest of thenetwork. It is precisely these groupings that our framework exports to theoperators.

We evaluate this framework by applying it to the configuration files offive production networks, including three universities and two private enter-prises. Through our empirical study we show that policy units capture usefulcharacteristics of a network’s policy and aid operators in discovering errors.

Demystifying Configuration Challenges and Trade-Offs in ISP Services:Although enterprises are rapidly adopting ISP services such as VPN and VPLS,there is little systematic work on the design challenges and trade-offs that ISPsface in providing them. We conducted a study [17] with the aim of understand-ing the complexity underlying the layer-3 design of services and highlightingpotential factors that hinder service introduction, evolution and management.

16

Using daily snapshots of configuration and device metadata collected from atier-1 ISP, we examined the logical dependencies and special cases in the deviceconfigurations for five different network-based services. We found: (1) thedesign of the data-plane’s core is usually service-agnostic and simple, but thecontrol-planes for different services become more complex as services evolve;(2) more crucially, the configuration of the service’s edge routers inevitablybecomes more complex over time, potentially hindering key management taskssuch as service upgrades and troubleshooting; and (3) there are key service-specific issues that also contribute significantly to the overall design complexity.

Thus, high and prevalent complexity can impede the evolution of suchservices in response to changes in an enterprise’s core requirements. We showinitial evidence that some of the complexity can be systematically mitigated. Weshow how the complexity in customer provisioning and service control-planedesign can be controlled by making an informed choice of vendors and bypicking appropriate control-plane design.

Understanding Data Center Traffic Characteristics: We conducted thelargest empirical study of three categories of data centers focusing on theirdesign and traffic properties. These data centers consist of university andenterprise campus data centers, data centers employed by large on-line ser-vice providers offering Internet-facing applications, and data-intensive (Map-Reduce style) data centers. To conduct this study, we collected and analyzedSNMP statistics, topology, and packet-trace information from these data cen-ters.

Our key findings are that: (1) data center traffic is bursty and differs signifi-cantly from WAN traffic; (2) bisection bandwidth does not play a significantrole in determining the performance of these data centers; and (3) centralizedmanagement platforms, with support from server-based modules, can helpachieve a variety of network-wide performance and cost objectives in these datacenters. With this study on data centers, we advance the state of the art in datacenter design by providing empirical insights and a deeper understanding ofthe characteristics of traffic within the context of data centers.

MicroTE: In our study of data centers, we observed differences between thetraffic distribution of data center traffic and WAN traffic. Given these differences,

17

it is natural to expect that the traffic engineering techniques designed for WANtraffic distributions will deliver suboptimal performance within the context ofdata centers.

In studying these traffic engineering techniques, we discovered that theyperformed 15% to 20% less than the optimal solution, where the optimal solutionis defined as an assignment of traffic to paths that produces the lowest MaximumLink Utilization (MLU). We observed that these techniques suffer due to thefollowing three drawbacks: (1) lack of multi-path routing, (2) lack of loadsensitivity, and (3) lack of a global view in making traffic engineering decisions.

To overcome these drawbacks, we designed and implemented MicroTE [19],a traffic engineering technique that adapts to traffic variations by leveraging theshort term and partial predictability of the traffic matrix. MicroTE tracks thetraffic matrix over time and uses this information to determine predictability aswell as to determine the hot-spots (a set of Top-of-Rack switches consistentlycommunicating over 50% of the unpredictable traffic). MicroTE reserves pathsfor the hot-spots and routes the predictable traffic around these reserved paths.We implemented MicroTE within the OpenFlow framework and with minormodification to the end hosts. In our evaluations, we show that our systemperforms close to the optimal solution and imposes minimal overhead on thenetwork, making it appropriate for current and future data centers.

1.5 how to read this thesis

In the remainder of this thesis, we present a novel set of metrics for capturingand quantifying the complexity underlying the management of network config-uration (Chapter 2), then we show (in Chapter 3) how one of the underlyingmodels for the metrics can be transformed into a framework, Policy Units, forexposing a network’s high level policies and goals. Next (in Chapter 4), wedevelop a strawman approach that illustrates how our complexity metrics canbe integrated into daily operations to mitigate complexity by guiding operatorstowards less complex protocols and design goals. With an eye towards per-formance within data centers, we shift from the network configuration to theprotocols being configured (Chapter 5 and 6). In Chapter 5 , we analyze and de-

18

velop models that characterize the traffic patterns resulting from the interactionbetween the applications and the configured routing protocols. Using theseinsights we propose a novel traffic engineering technique, MicroTE (Chapter 6),that improves performance while requiring minimal configuration from thenetwork operator. We conclude by presenting an overview of related worksin Chapter 7, and then summarizing our contributions and briefly alluding tofuture directions for research in Chapter 8.

19

2 measuring the complexity of enterprise configurationmanagement

In this chapter, we describe our work on designing a set of metrics to capturethe difficulty of operating and managing the configuration of an enterprisenetwork. In order to design these metrics, we began with a review of formaltraining materials for network engineers (e.g., [68, 74]) and interviews with theoperators of several networks to understand the tools and processes they use tomanage their networks. From these sources, we extracted the “best commonpractices” used to manage the networks. On the hypothesis that the use of thesepractices should be discernible in the configuration files of networks that usethem, we developed models and techniques that tie these practices to patternsthat can be automatically detected and measured in the configurations.

The remainder of this section describes the networks we studied, the bestcommon practices we extracted, and a tutorial summary of network configura-tion in enterprise networks. The next sections precisely define our metrics, themeans for computing them, and their validation.

2.1 data set: studied networks

We studied a total of seven networks: four university networks and three enter-prise networks, as these were the networks for which we could obtain configu-ration files. For three of the university networks and two of the enterprises, wewere also able to interview the network operators to review some of the resultsof our analysis and validate our techniques. Table 2.1 shows the key propertiesof the networks.

Type Number of Routers Number of End-hosts Interviewed?Univ-1 12 29,000 YUniv-2 19 9,000 NUniv-3 24 8,000 YUniv-4 36 26,000 YEnet-1 8 6,000 YEnet-2 83 N/A NEnet-3 19 5,000 Y

Table 2.1: Studied networks.

20

(a)0 500 1000 1500 2000

0

0.2

0.4

0.6

0.8

1

File Sizes (in Lines)

Univ−1Univ−2Univ−3Univ−4Enet−1Enet−2Enet−3

(b)

Figure 2.1: (a) Distribution of configuration file size across networks. (b) Frac-tion of configuration dedicated to configuring each aspect of router functional-ity.

Figure 2.1(a) plots the distribution of configuration file sizes for the networks.The networks cluster into three groups: Univ-2 and the enterprises consist ofrelatively small files, with 50% of their files being under 500 lines, while 90% ofthe files in Univ-1 and Univ-3 are over 1,000 lines and Univ-4 has a mix of smalland large files. As we will see, configuration file size is not a good predictorof network complexity, as Univ-2 (small files) is among the most complicatednetworks and Univ-3 (large files) among the simplest.

Figure 2.1(b) breaks down the lines of configuration by type. The networks

21

differ significantly in the fraction of their configurations devoted to Packetfilters, widely known as ACLs, and routing stanzas. Univ-1 and the enterprisesspend as many configuration lines on routing stanzas as on ACLs, while Univ-2,-3 and -4 define proportionately more ACLs than routing stanzas. Interfacedefinitions, routing stanzas, and ACL definitions (the key building blocks fordefining layer-3 reachability) account for over 60% of the configuration in allnetworks.

All the networks used some form of tools to maintain their configurations [1,41]. Most tools are home-grown, although some commercial products are in use.Most had at least spreadsheets used to track inventory, such as IP addresses,VLANs, and interfaces. Some used template tools to generate portions ofthe configuration files by instantiating templates using information from theinventory database. In the sections that follow, we point out where tools helped(and sometimes hurt) operators.

2.2 overview of network design and configuration

Based on our discussions with operators and our review of training materials,we extract the best common practices that operators follow to make it easier tomanage their networks. Our complexity metrics quantify how well a networkadheres to these strategies or, equivalently, to what extent a network deviatesfrom them.

Uniformity. To the extent possible, operators attempt to make their net-works as homogeneous as possible. Special cases not only require more thoughtand effort to construct in the first place, but often require special handling dur-ing all future network upgrades. To limit the number of special cases operatorsmust cope with, they often define a number of archetypal configurations, whichthey then reuse any time that special case arises. We call these archetypes roles.

Tiered Structure. Operators often organize their network devices into tiersto control the complexity of their design. For example, they may define somerouters to be border routers that connect with other networks, some routers tobe core routers that are densely connected, and the remaining routers as edgerouters that connect hosts.

22

Short Dependency Chains. Routers cannot be configured in isolation, asfrequently one part of the configuration will not behave correctly unless otherparts of the configuration, sometimes on other routers, are consistent with it.We define this to be a dependency between those configuration lines. Operatorsattempt to minimize the number of dependencies in their networks. This isbecause making a change to one configuration file but not updating all the otherdependent configurations will introduce a bug. Since the configurations do notexplicitly declare all their dependencies, operators’ best strategy is to minimizethe number of dependencies.

Overview of a Configuration FileAll our complexity metrics are computed on the basis of router configuration

files. Before defining the metrics, we describe the layout of the configuration filefor a network router and provide an overview of the mechanisms (e.g., routing,ACLs and VLANs) used when designing enterprise networks.

Figure 2.2: A sample configuration file.

The configuration file for a Cisco device consists of several types of stanzas(devices from other vendors have similar stanza-oriented configurations). Astanza is defined as the largest continuous block of commands that encapsulatea piece of the router’s functionality.

In Figure 2.2, we show a simple configuration file consisting of the three

23

most relevant classes of stanzas: interface in lines 1-3, routing protocol in lines 5-11, and ACL in lines 13-15. The behavior exhibited by a router can be explainedby the interactions between various instances of the identified stanzas.

Egress filtering, i.e., preventing local hosts from sending traffic with IPaddresses that do not belong to them, has become a popular way to combatIP-address hijacking. Networks implement egress filtering by defining a packetfilter for each interface and creating a reference to the appropriate ACL from theinterfaces. For example, line 3 exemplifies the commands an operator woulduse to setup the appropriate references.

The purpose of most layer-3 devices is to provide network-wide reachabilityby leveraging layer-3 routing protocols. Network-wide reachability can beimplemented by adding a routing stanza and making references between thatstanza and the appropriate interfaces. Lines 5-11 declare a simple routingstanza with line 8 making a reference between this routing protocol and theinterface defined earlier. Even in this simple case, the peer routing protocolstanza on neighboring devices must be configured consistent with this stanzabefore routes can propagate between the devices and through the network.

More complex reachability constraints can be imposed by controlling routedistribution using ACLs. Line 15 is a filter used to control the announcementsreceived from the peer routing process on neighboring routers.

VLANs are widely used to provide fine-grain control of connectivity, butthey can complicate configuration by providing an alternate means for packetsto travel between hosts that is independent of the layer-3 configuration. In atypical usage scenario, each port on a switch is configured as layer-2 or layer-3. For each layer-3 port there is an interface stanza describing its properties.Each layer-2 port is associated with a VLAN V . The switches use trunkingand spanning tree protocols to ensure that packets received on a layer-2 portbelonging to VLAN V can be received by every host connected to a port onVLAN V on any switch.

Layer-2 VLANs interact with layer-3 mechanisms via virtual layer-3 interfaces—an interface stanza not associated with any physical port but bound to a specificVLAN (lines 1–3 in Figure 2.2). Packets “sent” out of the virtual interface aresent out of the physical ports belonging to the VLAN, and packets received by

24

the virtual interface are handled using the layer-3 routing configuration.

2.3 implementation complexity: reference chains

As the above description indicates, enabling the intended level of reachabilitybetween different parts of a network requires establishing reference links in theconfiguration files of devices. Reference links can be of two types: those betweenstanzas in a configuration file (intra-file references) and those across stanzasin different configuration files (inter-file). Intra-file references are explicitlystated in the file, e.g. the links in line 8 (Figure 2.2) from a routing stanza toan interface, and in line 10 from a routing stanza to an ACL—these referencesmust be internally consistent to ensure router-local policies (e.g., ingress filtersand locally attached networks) are correctly implemented. Inter-file referencesare created when multiple routers refer to the same network object (e.g., aVLAN or subnet); these are central to configuring many network-wide functionsincluding, routing and reachability. Unlike their intra-file counterparts, not allinter-file references can be explicitly declared. For example, line 2 refers to asubnet which is an example of an entity that cannot be explicitly declared.

As our interviews with operators indicate (§2.3), in some networks thereference links must be manually established. In other networks, some of thereference links within a device are set using automated tools, but many of theinter-file references such as trunking a VLAN on multiple routers and settingrouting protocol adjacencies must be managed manually.

To quantify the complexity of reference links, we first construct a referentialdependency graph based on device configuration files. We compute a set of first-order complexity metrics which quantify the worst-case complexity of configuringreference links in the network. Because reference links often play a role inimplementing some network-wide functionality, we also define second-ordermetrics that estimate the overall complexity of configuring such functionality.We focus on routing in this discussion, as operators report that it is a significantconcern.

25

Complexity Model: Referential Dependence Graph

We use a two-step approach to parse configuration files and create a dependencygraph.

1. Symbol Table Creation. Router-vendor documentation typically lists thecommands that can appear within each configuration stanza and the syntax forthe commands. Based on this, we first create a grammar for configuration linesin router configuration files. We build a simple parser that, using the grammar,identifies “tokens” in the configuration file. It records these tokens in a symboltable along with the stanza in which they were found and whether the stanzadefined the token or referred to it. For example, the access-list definitions inlines 13-14 of Figure 2.2 define the token ACL 9 and line 3 adds a reference toACL 9.

2. Creating Links. In the linking stage, we create reference edges betweenstanzas within a single file or across files based on the entries in the symboltable. We create unidirectional links from the stanzas referencing the tokens tothe stanza declaring the tokens. Because every stanza mentioning a subnet orVLAN is both declaring the existence of the subnet or VLAN and referencingthe subnet/VLAN, we create a separate node in the reference graph to representeach subnet/VLAN and create bidirectional links to it from stanzas that mentionit.

We also derive maximal sub-graphs relating to layer-3 control-plane func-tionality, called “routing instances” [54]. A routing instance is the collectionof routing processes of the same type on different devices in a network (e.g.,OSPF processes) that are in the transitive closure of the “adjacent-to” relation-ship. We derive these adjacencies by tracing relationships between routingprocesses across subnets that are referenced in common by neighboring routers.Taken together, the routing instances implement control-plane functionality ina network. In many cases, enterprise networks use multiple routing instancesto achieve better control over route distribution and to achieve other admin-istrative goals [54]. For example, some enterprises place routes to differentdepartments into different instances—allowing designers to control reacha-bility by controlling the instances in which a router participates. Thus, it is

26

important to understand the complexity of configuring reference links thatcreate routing instances.

Complexity Metrics

We start by capturing the baseline difficulty of creating and tracking referencelinks in the entire network. The first metric we propose is the average configura-tion complexity, defined as the total number of reference links in the dependencygraph divided by the number of routers. This provides a holistic view of thenetwork.

We also develop three second-order metrics of the complexity of config-uring the Layer-3 control plane of a network. First, we identify the numberof interacting routing policy units within the network that the operator musttrack globally. To do this, we count the number of distinct routing instancesin the entire network. Second, we capture the average difficulty of correctlysetting each routing instance by calculating the average number of referencelinks per instance. Finally, we count the number of routing instances each routerparticipates in. In all three cases, it follows from the definition of the metricsthat higher numbers imply greater complexity for a network.

Evaluating Referential Chains: Insights From Operator Interviews

We derived referential complexity metrics for all seven networks. Our observa-tions are summarized in Table 2.2. Interestingly, we note that the referentialmetrics are different across networks for example, very low in the cases of Enet-1and much higher for Univ-1. For five of the seven networks, we discussed ourfindings regarding referential dependencies with network operators.

We present the insights we derived focusing on three key issues: (1) Val-idation: are the referential dependencies we inferred correct and relevant inpractice (meaning that these are links that must be created and maintained forconsistency and/or correctness)? (2) Complexity: are our complexity metricsindicative of the amount of difficulty operators face in configuring their net-works? (3) Causes: what caused the high referential complexity in the networks

27

(where applicable)?

Network Average referential Layer-3 functionality Interviewed?complexity per router Number of routing Complexity Instances

instances per instance per routerUniv-1 41.7 14 35.8 2.5 YUniv-2 8.3 3 58.3 1.1 NUniv-3 4.1 1 99 1 YUniv-4 75 2 902 1 YEnet-1 1.6 1 16 0.7 YEnet-2 7.5 10 62 1.2 NEnet-3 22 8 52 1.4 Y

Table 2.2: Complexity due to referential dependence. Networks where wevalidated results are marked with a “Y.”

Validation. We showed each network’s referential dependence graph tothe network operators, along with sub-graphs corresponding to the routingprotocol configuration in their network. All operators confirmed that the classesof links we derived (e.g., between stanzas of specific kinds within routers andacross routers) were relevant. We also gave operators tasks involving changesto device configurations (specifically, add or remove a specific subnet, applya new filter to a collection of interfaces). We verified that our reference linkstracked the action they took. These two tests, while largely subjective, validatedour referential dependency derivation.

As an aside, the dependency graph seems to have significant practical value:Univ-1 and Enet-1 operators felt the graph was useful to visualize their networks’structure and identify anomalous configurations.

Do the metrics reflect complexity? Our second goal was to test if themetrics tracked the difficulty of maintaining referential links in the network.To evaluate this, we gave the operators a baseline task: add a new subnet ata randomly chosen router. We measured the number of steps required andthe number of changes made to the routing configuration. This is summarizedbelow.

Network Num steps Num changes to routingUniv-1 4-5 1-2Univ-3 4 0Enet-1 1 0

28

In networks where the metrics are high (Table 2.2), operators needed moresteps to set up reference links and needed to modify more routing stanzas. Thus,the metrics appear to capture the difficulty faced by an operator in ensuringconsistent device-level and routing-level configuration. We elaborate on thesefindings below.

In Univ-1, the operators used a home-grown automated tool that generatesconfiguration templates for adding a new subnet. Thus, although there aremany references to set, automation does help mitigate some aspects of thiscomplexity.

Adding the subnet required Univ-1’s operator to modify routing instancesin his network. Just as our second-order complexity metrics predicted, thistook multiple steps of manual effort. The operator’s automation tool actuallymade it harder to maintain references needed for layer-3 protocols. Note thatper Table 2.2, an average Univ-1 router has two routing instances present on it.These are a “global” OSPF instance present on all core routers and a smallerper-router RIP instance. The RIP instance runs between a router and switchesdirectly attached to the router, and is used to distribute subnets attached tothe switches into OSPF. On the other hand, OSPF is used to enable globalreachability between subnets and redistribute subnets that are directly attachedto the router. When a new subnet is added to Univ-1, the operator’s toolautomatically generates a network command and incorporates it directly intothe OSPF instance. When the subnet needs to be attached to a layer-2 switch,however, the network statement needs to be incorporated into RIP (and notOSPF). Thus, the operator must manually undo the change to OSPF and updatethe RIP instance. Unlike the OSPF instance, the network statements in RIPrequire parameters that are specialized to a switch’s location in the network.

Univ-3 presents a contrast to Univ-1. The operator in Univ-3 required 4steps to add the subnet and this is clearly shown by the first-order complexitymetric for Univ-3. In contrast to Univ-1, however, almost all of the steps weremanual. In another stark difference from Univ-1, the operator had no changesto make to the routing configuration. This is because the network used exactlyone routing instance that was set-up to redistribute the entire IP space. Thissimplicity is reflected in the very low second-order metrics for Univ-3.

29

The operator in Enet-1 had the simplest job overall. He had to perform onestep: create an interface stanza (this was done manually). As with Univ-3, therouting configuration required little perturbation.

In general, we found that the metrics are not directly proportional to thenumber of steps required to complete a management task like adding a subnet,but the number of steps required is monotonically increasing with referential complex-ity. For example, Univ-1 with a reference metric of 41.75 required 4 - 5 stepsto add a subnet. Univ-2, with a metric of 4.1 needed 4 steps and Enet-1 with ametric of 1.6 needed just one step.

Causes for high complexity. The most interesting part of our interviewswas understanding what caused the high referential complexity in some net-works. The reasons varied across networks, but our study highlights some ofthe key underlying factors.

The first cause we established was the impact of a network’s evolution overtime on complexity. In Univ-1, approximately 70% of reference links arose dueto “no passive interface” statements that attempt to create routing adjacenciesbetween neighboring devices. Upon closer inspection, we found that a largenumber of these links were actually dangling references, with no correspondingstatement defined at the neighboring router; hence, they played no role in thenetwork’s routing functionality. When questioned, the operator stated that thecommands were used at one point in time. As the network evolved and deviceswere moved, however, the commands became irrelevant but were never cleanedup.

The high second order complexity in Univ-1 results from an interestingcause: optimizing for monetary cost rather than for reducing complexity. Univ-1’s operator could have used a much smaller number of routing instances(e.g., a single network-wide OSPF) with lower referential counts to achieve thegoal of spreading reachability information throughout the network. However,according to the operator, using OSPF on a small number of routers, and RIPbetween switches and routers, was significantly cheaper as OSPF-licensedswitches cost more. Hence this routing design was adopted although it wasmore complex.

Sometimes, the policies being implemented may require high referential complex-

30

ity. For instance, Univ-3 imposes global checks for address spoofing, andtherefore applies egress filters on all network interfaces. These ACLs accountedfor approximately 90% of the links in the dependency graph. Similarly, Univ-4uses ACLs extensively, resulting in high referential complexity. Despite similarunderlying causes, Univ-4 has a higher complexity value than Univ-3 becauseit employs significantly more interfaces and devices.

2.4 implementation complexity: router roles

When creating a network, operators typically start by defining a base set ofbehaviors that will be present across all routers and interfaces in the network.They then specialize the roles of routers and interfaces as needed to achievethe objectives for that part of the network, for example, adding rate shaping todorm subnets, and additional filters to protect administrative subnets.

Designers often implement these roles using configuration templates [21].They create one template for each role, and the template specifies the config-uration lines needed to make the router provide the desired role. Since theconfiguration might need to be varied for each of the routers, template systemstypically allow the templates to contain parameters and fill in the parameterswith appropriate values each time the template is used. For example, the tem-plate for an egress filter might be as shown in Figure 2.3, where the ACL restrictspackets sent by interface III to those originating from the subnet configured tothe interface. The designer creates specific configuration stanzas for a routerby concatenating together the lines output by the template generator for eachbehavior the router is supposed to implement.

Figure 2.3: Example of a configuration template.

From a complexity stand-point, the more base behaviors defined within thenetwork, the more work an operator will have to do to ensure that the behaviorsare all defined and configured correctly and consistently. Further, the greater

31

the degree of specialization required by routers to implement a template role,the more complex it becomes to configure the role.

We show how to work backwards from configurations to retrieve the originalbase behaviors that created them. By doing so, we can measure two key aspectsof the difficulty of configuring roles on different routers in a network: (1)How many distinct roles are defined in the network? (2) How many routersimplement each role?

Complexity Model: Copy-Paste Detection

We identify roles that are "shared" by multiple routers using a copy-paste detec-tion technique. This technique looks for similar stanzas on different routers.

We build the copy-paste detection technique using CCFinder [44], a toolthat has traditionally been used to identify cheating among students by lookingfor text or code that has been cut and pasted between their assignments. Wefound that CCFinder by itself does not identify templates of the sort used inrouter configuration (e.g., Figure 2.3). To discover templates, we automaticallypreprocess every file with generalization. Generalization replaces the commandarguments that may vary with wildcard entries—for example, IP addresses arereplaced by the string "IPADDRESS." Our implementation uses the grammar ofthe configuration language (Section 2.3) to identify what parameters to replace.

Complexity Metrics

Our first metric is the number of base behaviors defined within the network. Wedefine a base behavior as a maximal collection of shared-template stanzas thatappear together on a set of two or more routers. As the number of base behaviorsincreases, the basic complexity of configuring multiple roles across networkrouters increases.

To compute the number of base behaviors, we first identify the shared-template device set of each template—this is the set of devices on which theconfiguration template is present. Next, we coalesce identical sets. To elaborate,we write the device set for a shared-template stanza as STi = {Di1,Di2, . . . ,Diki}

32

where theDij represents a router that contains a configuration stanza generatedfrom shared template i. We scan the shared-template device sets to identifyidentical sets: if two shared-template stanzas are present on the same set ofrouters, then the stanzas can be considered to have arisen from a single, largertemplate; the stanzas are merged and one of the sets discarded. The finalnumber of distinct device sets that remains is the number of base behaviors.

As a second-order metric, we quantify the uniformity among devices interms of the behaviors defined on them. If all devices in the network exhibitthe same set of behaviors (i.e., they all have the same shared template), thenonce an operator understands how one router behaves, it will be easier for himto understand how the rest of the routers function. Also, updating the roles issimple, as all routers will need the same update.

To measure uniformity, we compute the median and mean number of de-vices in the device sets. We evaluated other information-theoretic metrics, suchas entropy. However, as our empirical study will show, these simple metrics, to-gether with the number of base behaviors, suffice to characterize the behaviorsdefined in a network.

Evaluation of Router Roles: Insights from Operator Interviews

Like the referential metrics, we validated our role metrics through interviewswith five operators. For this discussion, we focus on the use of ACLs, andTable 2.3 shows the role metrics for each network. We also evaluated rolesacross the entire configuration file, and the results are consistent with those forACLs.

Validation. When shown the shared templates extracted by our system,each of the operators immediately recognized them as general roles used intheir networks and stated that no roles were missed by our technique. Forexample, Univ-1 operators reported seven roles for ACLs in their network: onerole for an SNMP-related ACL, one role for an ACL that limits redistributionof routes between OSPF and RIP (these first two roles are present on mostrouters), and five ACLs that filter any bogus routes that might be advertised bydepartmental networks connected to the university network, one for each of

33

Network Number of Routers Shared template behaviors Interviewed?Number Device set size

Median MeanUniv-1 12 7 2 4.43 YUniv-2 19 19 2 5.75 NUniv-3 24 10 2 8.3 YUniv-4 24 28 3.5 4.3 YEnet-1 10 1 2 2 YEnet-2 83 5 3 34.2 NEnet-3 19 6 7.5 8.8 Y

Table 2.3: Roles extracted from ACLs.

the five departments (these are found only on the routers where the relevantnetworks connect).

Enet-3 has separate templates for sub-networks that permit multicast andthose that do not, as well as templates encoding special restrictions appliedto several labs and project sub-networks. Enet-1, the network with the fewestshared templates, has a pair of core routers that share the same set of ACLs.The remaining ACLs in the network are specific to the special project subnetsthat are configured on other routers. Univ-4, the network with the most sharedtemplates, has so many roles that it uses multiple different types of egress filters,each of which is applied to a subset of the routers. There are also several specialcase requests from various departments, each represented as an ACL appliedto 2 - 3 routers.

Do the metrics reflect complexity? The relationship between number ofroles and the complexity of the network is indicated by the types of tools andwork process used by the operators.

Operators of the network with the fewest roles, Enet-1, modify all the ACLsin their network manually—they are able to manage without tools due to theuniformity of their network. Operators at Univ-1 have tools to generate ACLs,but not to track relationships between ACLs, so they push all ACLs to all routers(even those that do not use the ACL) in an effort to reduce the complexity ofmanaging their network by increasing the consistency across the configurationfiles (our shared template system was programmed to ignore ACLs that are notused by the router: this explains why the mean device set size is not larger for

34

Univ-1). The environment at Univ-3 is similar to Univ-1, with roughly the samenumber of ACL roles and similar tools that can create ACLs from templates, butnot track relationships between them. The Univ-3 operators took the oppositeapproach to Univ-1, pushing each ACL only to the routers that use it, but usingmanual process steps to enforce a discipline that each ACL contain a commentline listing all the routers where an instance of that ACL is found. Operatorsthen rely on this meta-data to help them find the other files that need to beupdated when the ACL is changed.

Causes of high complexity. In general, the number of shared templateswe find in a network directly correlates with the complexity of the policies theoperators are trying to realize. For example, Univ-1’s goal of filtering bogusroute announcements from departmental networks requires applying a controlplane filter at each peering point. Similarly, Univ-4 has policies defining manydifferent classes of subnets that can be attached to the network, each one needingits own type of ACL (e.g., egress filtering with broadcast storm control andfiltering that permits DHCP). There is no way around this type of complexity.

Interestingly, the number of roles found in a network appears to be largelyindependent of the size of the network. For example, Enet-2 and Enet-3 havethe same number of roles even though they differ greatly in size. Rather, thenumber of roles seems to stem directly from the choices the operators made indesigning their networks, and how uniform they chose to make them.

2.5 inherent complexity

A network’s configuration files can be viewed as the “tools” used by networkoperators to realize a set of network-wide reachability policies. These policiesdetermine whether a network’s users can communicate with different resourcesin the network (e.g other users or services). The policies that apply to a usercould depend on the user’s “group,” her location, and other attributes.

The reachability policies fundamentally bound an operator’s ability to em-ploy simple configurations network-wide. Consider a network with a “simple”reachability policy, such as an all-open network that allows any pairs of usersto have unfettered communication, or at the opposite end of the spectrum, a

35

network where all communication except those to a specific set of servers is shutoff. Such policies can be realized using fairly simple network configurations.On the other hand, for networks where the reachability policies are complex,i.e., where subtle differences exist between the constraints that apply to differentsets of users, implementing the policies will require complex configuration.

We develop a framework for quantifying the complexity of a network’sreachability policies. We refer to this as the network’s inherent complexity. Weuse feedback from operators to both validate our metrics and understand thefactors behind the inherent complexity (where applicable). Ultimately, we wishto tie inherent complexity back to the configuration complexity and examinethe relationship between the two. We discuss this in §2.8.

To derive inherent complexity, we first derive the static reachability betweennetwork devices, which is the set of packets that can be exchanged betweenthe devices. We also refer to this as the reachability set for the device pair. Ourinherent complexity metrics essentially quantify the level of uniformity (or thelack of it) in the reachability sets for various paths in a network.

2.6 complexity model: reachability sets

For simplicity, we assume that network routers have IP subnets attached toth