Institut fürTechnische Informatik undKommunikationsnetze

Martin Müller

Integration of Measurement Probesinto a Distributed Measurement Plane

Semester Thesis SA-2015-28October 2015 to January 2016

Supervisors: Brian Trammell, Mirja KühlewindProfessor: Prof. Dr. Laurent Vanbever

1

Abstract

The internet is based on distributed protocols and decentralized authorities, making the designresilient to complete failure, but in turn also making locating single points of failure harder. Toaid in mitigating this problem, multiple large-scale measurement schemes have been created,among them mPlane and RIPE Atlas.In this thesis, a wrapper around RIPE Atlas for mPlane is presented, to provide access to theRIPE Atlas infrastructure through the mPlane protocol. This is then used to perform extensivetraceroute measurements to gain more insight into the network structure by visualizing the re-sults.

Contents

1 Introduction 31.1 The Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Background 52.1 mPlane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.1 Protocol Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.1.2 Generic Component and Client . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2 RIPE Atlas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Design and Implementation 83.1 Creating RIPE Atlas Measurements . . . . . . . . . . . . . . . . . . . . . . . . . 83.2 Getting RIPE Atlas Measurement Results . . . . . . . . . . . . . . . . . . . . . . 103.3 Additions to the Reference Implementation . . . . . . . . . . . . . . . . . . . . . 12

3.3.1 Improvements to mpcli . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.3.2 Components Returning Different Receipts . . . . . . . . . . . . . . . . . . 13

4 Evaluation 144.1 Visualization Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.2 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5 Summary and Outlook 19

A Additions to mPlane 20A.1 New Registry Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20A.2 New Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

A.2.1 Creating Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21A.2.2 Getting Measurement Results . . . . . . . . . . . . . . . . . . . . . . . . . 22

A.3 New mpcli commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

B Visualization Tool Usage 24

C Declaration of Originality 25

List of Abbrevations 26

List of Figures 27

List of Tables 28

Bibliography 29

2

Chapter 1

Introduction

The internet’s design is based on distributed protocols and algorithms, and has no central au-thority. This makes it resilient to attacks and failures from the perspective of the entire system,but makes it very hard to identify the source of errors in small scales. A single middlebox in anetwork could for example modify every packet’s TTL field it forwards. To detect this change, onewould need access to both a sending and a receiving point, and knowledge of the path takenthrough the network. To locate where this change takes place, you need even more access tothe network, e.g. send packets between every two points on the route originally taken. A real,similar example of this would be ECN enabled connections. ECN is an extension to TCP and IPto signal congestion, and uses previously unused fields in both headers. A study made in 2014found that connections to about 0.82% of the top one million websites fail if ECN is enabled [1]when accessing from certain parts on the world, but work fine from others. This happens eventhough ECN support of connections should be completely transparent to middleboxes that donot know about ECN.Locating middleboxes by sending packets between every two points in the network is possiblein small networks, but not so on a bigger or even global scale, where you most likely only haveaccess to either the sender or the receiver, and nothing in between. Another problem is thatall measurement going beyond just the RTT are very error prone: Paths through the internetare in constant flux and could change any time during e.g. a traceroute measurement. To com-bat these problems, there are multiple large-scale measuring infrastructures available to probethe internet and provide the data necessary to locate failures, among them are mPlane[2] andRIPE Atlas[3]. A large amount of accessible measuring points provide data from many differentlocations in the network in enough quantity to offset their unreliability. mPlane was used to findthe broken ECN connections mentioned earlier, so it is desirable to access the big measuringinfrastructure that is RIPE Atlas with mPlane for easy interoperability between more specializedand large-scale measurements. The idea is that the large amount of measurement data fromRIPE Atlas can be used to locate middleboxes that break ECN connections.

1.1 The Task

The goals of this thesis can be split up in roughly two parts:

• Write a wrapper for mPlane to access the RIPE Atlas measuring infrastructure throughthe mPlane protocol. mPlane is more generic regarding the measurements possible, butlacks the availability of large amounts of publicly accessible probes around the world.

• Use this wrapper to collect large amounts of traceroute measurement data to one singlepoint for better understanding of the network. Originally, the problems found in [1] were tobe investigated, but when rerunning the tests in January 2016, the known broken websitesfrom 2014 did not show any connection problems. Because of this, the traceroute data isnow used to visualize the network and gain more general insights into its structure.

3

1.2 Overview 4

1.2 Overview

Chapter 2 explains mPlane’s and RIPE Atlas’ structure in as much detail as necessary for thisthesis. Chapter 3 details the design of the RIPE Atlas wrapper for mPlane. Chapter 4 uses thewrapper to collect large amounts of traceroute data to visualize the network around a targetpoint. Chapter 5 presents a short summary and an outlook for future work.

Chapter 2

Background

mPlane and RIPE Atlas are measuring platforms to perform measurements in the internet. Whilethey have the same goals, their execution differs greatly. mPlane is explained in more detail thanRIPE Atlas, as the implementation of chapter 3 is more deeply involved with mPlane.

2.1 mPlane

Rather than providing an infrastructure for measurements, mPlane is a protocol, based on HTTP.Figure 2.1a shows a simplified model of its structure. There are two main elements: clients andcomponents. Components execute measurements based on a client’s orders, and the clientthen collects those results. Note that the connection is direct; Both client and component needto know how to reach each other, and there’s no central registry to find components providingmeasurements. A client can connect to multiple components, and a component can connect tomultiple clients. The components are free to implement measurements of any type, mPlane onlydefines how they exchange information with each other.

2.1.1 Protocol Operation

The communication between clients and components consists of different messages of differenttypes. Each type can only be sent from either a client or a component, but not from both. Figure2.2 shows a graphical representation of the communication. Note that this is a simplified model,it can be more sophisticated if so desired.At the very beginning, after client and component established a connection, the componentsends capabilities, describing what the component is capable of, to the client. They consist ofa list of parameter and result names, where parameters can be constrained to either a set ora range of values. Parameters represent the customizable bit of the measurement and are re-quired to run the measurement, while the result names tell the client what kind of results it canexpect. The names and their type are predefined in a registry, but can be extended at will. Tokeep capabilities apart from each other easily, a hash is calculated over all parameter and resultnames, called a token.If a client wants to start a measurement, it sends a specification to the component. A speci-fication is almost the same as a capability, but parameters have values now. Again a hash iscalculated to quickly compare specifications, but this time the values of the parameters get in-cluded.As soon as a component receives a specification, it will send back a receipt and start operation.The receipt can be used by the client to retrieve the results at a later date. It is also an indicationthat the component has not yet finished its tasks. The client turns the receipt into a redemptionand sends it back to the component. The tokens for both receipt and redemption are the sameas the one for the specification.When a component receives a redemption, it can either return a receipt again if it’s not doneyet, or return a result. The result is again very similar to a capability, but this time additionally tothe parameters, the results have values too. The results are laid out in a table: A result namedefines a column, and an arbitrary amount of rows represent the results. The token of the result

5

2.2 RIPE Atlas 6

(a) mPlane (b) RIPE Atlas

Figure 2.1: Structure of mPlane and RIPE Atlas

(a) The component announcescapabilities

(b) The client invokes a specifi-cation and gets a receipt in re-turn

(c) The client redeems a receiptand gets a result or another re-ceipt

Figure 2.2: Messages exchanged between client and component in mPlane

is the same as the specification’s token.The messages’ tokens are implementation defined, but the reference implementation1 imple-ments them as described here, and are also used like that in the implementation in chapter 3.A more detailed protocol specification can be found in the GitHub of the reference implementa-tion2.

2.1.2 Generic Component and Client

The reference implementation includes a generic client and component. They provide a basicinterface to the protocol, without having to deal with the specifics. The client is a command-lineinterface, providing a way for humans to interact with components. The generic component con-sists of different so-called services, which provide a capability. A service is a single function thattakes a specification and returns a result. The component handles everything else, e.g. sendingreceipts. This allows for quick implementation of services that don’t need specific control overthe messages and are content with simple input/output relationships between specifications andresults.

2.2 RIPE Atlas

RIPE’s Atlas takes a different approach than mPlane. Instead of defining a protocol to be usedby highly customizable elements, it provides one central hub to gain access to over 9000 probes.Figure 2.1b shows how this is organized.Probes, small hardware devices, can be installed anywhere with internet access. They will thenconnect to the RIPE Atlas network, and provide the possibility to run specific measurement fromthat location. Because they are hardware probes, these measurements are predefined, and onlyparameters like time-out, target, or protocol can be changed by the user. Every probe providesinformation about its AS and IP prefix among others, and also gets a unique ID.To access these probes, there is a web interface at https://atlas.ripe.net. Users canrequest probes based on AS number, IP prefix, and more, and then schedule measurements.Measurements also have a unique ID, and can be public or private. For programmatic access,

1https://github.com/fp7mplane/protocol-ri2https://github.com/fp7mplane/protocol-ri/blob/master/doc/protocol-spec.md

2.2 RIPE Atlas 7

a RESTful interface and official Python packages3,4 are available. The RESTful API is used bybuilding specific URLs for the atlas.ripe.net subdomain and exchanging HTTP messagescontaining JSON data, but the official Python packages will be used in chapter 3 for more con-venient access. Only registered users can start new measurements, so a key generated by theuser has to be supplied to authorize these request. Furthermore, to access private measure-ments, another key has to be present in the request.RIPE Atlas provides access to a big measuring infrastructure, but lacks the flexibility of mPlane,while mPlane is very flexible, but has no publicly accessible widely distributed components. Thismakes building a mPlane component for RIPE Atlas a worthwhile investment, as the rather spe-cialized measurements made by mPlane components can be supplemented with more generalmeasurements from a high amount of vantage points.

3https://github.com/RIPE-NCC/ripe-atlas-cousteau4https://github.com/RIPE-NCC/ripe.atlas.sagan

Chapter 3

Design and Implementation

Because RIPE Atlas works conceptionally like mPlane, i.e. you start a measurement and thenwait for results, it was not necessary to create a mPlane component from scratch. Instead,two services for the generic component of the reference implementation were written. Oneservice creates new measurements, and the other queries for results. The services were splitup to enable querying the RIPE Atlas measurement database independently of starting newmeasurements and are described in the next two sections. Because there were some changesnecessary to the mPlane implementation, there is a short summary of those in section 3.3. Theservices created can handle ping measurements, because they are very easy to implement,and traceroute measurements, because they are needed for chapter 4. The probes supportparis traceroute[4], an improvement over the traditional traceroute, so parameters related to thatare also made available in mPlane. Extending the services for other measurement types shouldnot be very hard, as traceroutes have the most complicated result structure and all conceptsfrom there can be reused. The only difficulty lies in mapping RIPE Atlas parameters to mPlaneparameters. The services only support IPv4, but adding IPv6 should not be hard, either.See Appendix A for a complete list of all additions to mPlane from the protocol’s perspective.

3.1 Creating RIPE Atlas Measurements

Creating measurements seems straightforward at first, but is not as easy as it seems. RIPEAtlas exposes a lot of parameters for customization, and the difficulty here is to decide whichparameters should be available in the mPlane specification. Because mPlane doesn’t haveoptional parameters or default values, every parameter has to be set every single time a speci-fication is created, which makes usability worse. Table 3.1 shows, which RIPE Atlas parametershave been made available in the mPlane component for a traceroute measurement. All otherparameters for RIPE Atlas have default values predefined on RIPE Atlas’ side. The parame-ters for the ping measurement are mostly the same, but without ripeatlas.protocol andripeatlas.paris.While most parameters have a direct correspondence, there are two special cases, namely themPlane when and the RIPE Atlas probes parameters. In both cases, one parameter in oneprotocol doesn’t exists and is not directly implementable in the other protocol. To overcomethis, the parameters get split up into multiple parameters while still providing the same options.The mPlane parameter when is a good example of this: RIPE Atlas does not have one singleparameter describing the timing of the measurement, but four of them. Luckily, mPlane’s whenencodes the same information as those four parameters, so translation between them is simple.The only difference is in the handling of one-off measurements, i.e. measurements that donot repeat over a longer span of time. mPlane requires either a measurement interval to bepresent or not in when’s constraint, and does not allow both at the same time. So from a user’sperspective, an interval always has to be specified, but will be ignored if when contains onlyone point in time. This way, both one-off and continuous measurements are possible, while alsoenforcing the minimum RIPE Atlas measurement interval of one minute.The other example of no direct correspondence of parameters is RIPE Atlas’ probes parameter.This is a list of probes that should actually execute the measurement. A user can select these

8

3.1 Creating RIPE Atlas Measurements 9

RIPE Atlas mPlane

Parameter Parameter Name Parameter Type Constraint Comments

protocol ripeatlas.protocol string ICMP,UDP,TCP

description ripeatlas.description string ∗ Automatically gen-erated if empty

paris ripeatlas.paris natural 0 ... 64 Zero disables paristraceroute

target destination.ip4 address ∗af 4 Only IPv4 is sup-

ported

interval

when time now ... future / 1mis_oneoff is true ifstart_time andend_time are equal

start_time

stop_time

is_oneoff

probes

ripeatlas.probe_id natural [∗] All probes from thespecifiedmeasurement getreused

ripeatlas.msm_id natural [∗]

ripeatlas.probe_source string ∗

Table 3.1: Assignment of RIPE Atlas parameters to mPlane parameters when creating mea-surements. "∗" means no constraints, "[ ]" means no or multiple values possible

probes with six methods:

• Area, one of ww (worldwide), west, north-central, south-central, north-east, or south-east

• Country, a two-letter country code

• Prefix, a IP prefix

• Autonomous System, a AS number

• Probe ID, a list of probe IDs

• Measurement, a list of past measurements to reuse probes from

A measurement creation request can contain an arbitrary amount of these probe selections, nomatter the selection method. This proves very challenging for mPlane, because as mentionedbefore, there are no optional parameters, so you’d need to specify every method every time,even though you only want one. Another problem is that additionally to the information fromwhere you want to select the probes, it also must be specified how many probes are requested,resulting in a pair of selection method and probe amount. This is also something mPlane doesn’thave, and a naive implementation would result in two mPlane parameters per method allowingmultiple values, one for the selection and one for the amount. The capability for such a servicewould then become more complicated, even more parameters would have to be introduced,resulting in an overly complex process. The implemented solution to this problem can be seenin table 3.1. The three mPlane parameters ripeatlas.probe_id, ripeatlas.msm_id, andripeatlas.probe_source provide the same functionality as the single RIPE Atlas probesparameter. ripeatlas.probe_id is a list of probe IDs, corresponding directly to the methodof selecting probes by ID. ripeatlas.msm_id is a list of measurement IDs, corresponding tothe method of selecting probes from a past measurement. The amount of probes is automat-ically set to the correct amount to reuse all probes. These two parameters can have multiplevalues and are implemented as a list, so as a side effect they can also be empty, emulating

3.2 Getting RIPE Atlas Measurement Results 10

Selection Method Token Syntax/Example

Area WW, West, North-Central, South-Central, North-East, South-East

Country CH

IP Prefix 1.2.3.4/16

AS Number ASN1234

Example ASN2133 10 US 2 South-East 20 47.25.0.0/15 42

Table 3.2: String tokens when requesting RIPE Atlas probes summarized

Result Column Type

delay.twoway.icmp.us.min natural

delay.twoway.icmp.us.mean natural

delay.twoway.icmp.us.50pct natural

delay.twoway.icmp.us.max natural

delay.twoway.icmp.us.count natural

source.ip4 address

destination.ip4 address

ripeatlas.probe_id natural

time time

Table 3.3: Result columns and their types in a ping result. Every result row is one measurementby one probe

optional parameters. Because the probe amount can be figured out programmatically for bothselection methods, only one parameter each is sufficient.The most interesting parameter is ripeatlas.probe_source, a string providing the possibil-ity to select probes by area, country, IP prefix, or AS number. The string consists of a series ofspace-separated tokens, interpreted in pairs. The first token specifies a method of selection andthe following one is a number specifying how many probes should be requested. The tokensare straightforward and summarized in table 3.2. Note that this method implies an even amountof tokens. A minor disadvantage of this method is that the client can’t know if the string hasthe correct syntax, only the component that parses the string can know if something is wrong.One could imagine creating a regular expression as a parameter constraint, but the expressionwould be extremely long and complicated, because it would have to e.g. include all two lettercountry codes. Also, mPlane does not support regular expressions as a parameter constraint.The result of this capability is a single ripeatlas.msm_id, the measurement ID returned byRIPE Atlas. For convenience, the service returns not a result but a receipt for the service forretrieving RIPE Atlas results (see section 3.2). The client was extended to automatically send aspecification to this service based on the receipt. For more details, see section 3.3.

3.2 Getting RIPE Atlas Measurement Results

RIPE Atlas’ ping measurements are simple and map well to the mPlane protocol. All necessaryparameters are already present, and one row in a result is one ping measurement by one probe.One additional parameter ripeatlas.probe_id has been introduced to provide informationover which result row comes from which probe. The only parameter besides the mandatorywhen is ripeatlas.msm_id. Table 3.3 shows all parameters present in a ping result.For traceroute measurements, again the only two parameters are when andripeatlas.msm_id. It gets more difficult when it comes to actual results, however. Atraceroute consists of a number of hops, representing the traversed IP addresses until a

3.2 Getting RIPE Atlas Measurement Results 11

Result Column Type Comments

intermediate.ip4 address The IP address of the hop

ripeatlas.traceroute_id natural Uniquely identifies the traceroute. In essence a hashof time and ripeatlas.probe_id

ripeatlas.hop_index natural Identifies the hop in the traceroute

ripeatlas.paris_id natural

ripeatlas.protocol string

hops.ip natural The total amount of hops

rtt.ms natural The round trip time of the packet

source.ip4 address IP address of the probe

destination.ip4 address

ripeatlas.probe_id natural

time time Time when measurement started

Table 3.4: Result columns and their types in a traceroute result. Every result row correspondsto a packet of a hop in a traceroute

destination is reached. Packets get sent with increasing TTL numbers, starting at one. Typically,multiple packets with the same TTL value get sent. mPlane’s method of storing results in tablescan be made to work with single traceroutes and only one packet per hop, by assigning onehop per row. But this is not feasible with RIPE Atlas traceroute results for at least two reasons:One, traceroute measurements have almost always more than one packets per hop to combatlost packets and get better results. Two, RIPE Atlas results contain much more than only onetraceroute result. The entire point of RIPE Atlas is to have a lot of probes available for mea-surements, so tens, hundreds, or even thousands of results are to be expected. One possibilityto deal with that in mPlane is to return multiple results, one for each trace. mPlane can savemultiple messages in so-called envelopes, and then transmit those. However, the expectationwhen sending a specification to a component is to get one result, not an envelope with lots ofthem. Moreover, envelopes are intended to be used for self-repeating measurements, wheremultiple results are expected. Due to these reasons, another method to deal with hundreds ofresults was implemented.The method implemented in the service unrolls all packets, hops, and traces. A traceroute resultfrom RIPE Atlas is a list of traces, which are lists of hops, which are lists of packets. The serviceturns these lists into rows in one and the same table. One row corresponds to one packet, andtwo different parameters are used to assign them to a specific hop in a specific trace. Figure 3.1shows how the unrolling works conceptually, table 3.4 shows how the result table looks like andhow it should be interpreted. A disadvantage is that for further analysis on the client’s side, thetable most likely has to be reconverted into lists again. This inconvenience is not big though,and is overshadowed by the fact that theoretically infinite results can be returned this way,despite the restrictive table form of mPlane results. As an example, traceroutes from 22 probesevery five minutes over eight hours resulted in 2,112 traces, producing 108,954 rows in themPlane result. If a particular hop did not respond, it is signified with the IP address 0.0.0.0and most other values at zero.A few comments for the when parameter are also necessary. This parameter can be interpretedhere as the time period from where you want results from, so the obvious constraint is past... now. This poses problems though, because the service for creating measurements createsa receipt for this service, and of course the receipt’s when lies in the future because you can’tstart new measurements in the past. So the actual constraint for when is past ... future.When the stop time of when is in the future, the service will wait until that time and then continue.To make sure RIPE Atlas had enough time to finish the measurements, the service always waitsuntil one minute has passed since the end time in the when parameter.

3.3 Additions to the Reference Implementation 12

Figure 3.1: Depiction of how traceroute lists get unrolled. Each traceroute is a list of hops, whichis a list of packets. Unrolled, this results in a list of packets. Every few packets build a hop, andevery few hops build a traceroute

3.3 Additions to the Reference Implementation

In the process of writing the services, several changes have been made to the reference im-plementation. A few bugs were fixed, but there were also some bigger changes and additionsmade.

3.3.1 Improvements to mpcli

mpcli is the generic client for mPlane mentioned in section 2.1.2, presenting itself as a com-mandline interface. It is possible to create specifications based on capabilities, and view results.This is convenient for quick testing of simple capabilities, but gets tedious when there are capa-bilities with a large amount of parameters. To improve this situation, it is now possible to storespecifications to a file. At a later time, these specifications can be loaded back into the client inthe form of parameter values, or optionally directly sent to a component. Specifications savedin the file are identified by their labels. It is possible to save multiple specifications to the samefile, but they have to have different labels. When loading a specification from a file, if there aremultiple specifications with the same label, only the first one will be loaded, and it’s not possibleto load a different one. Only the when parameter will not be loaded and has to be supplied everytime by the user. In the same vein, it’s possible to save results to a file. See section A.3 for a listof all new commands.An additional, minor improvement was made concerning parameters accepting multiple values.This addition was made just recently and is still experimental, so the commandline did not han-dle these correctly. Now, a string with space separated values will be interpreted as a list ofvalues if the parameter allows multiple values.

3.3 Additions to the Reference Implementation 13

3.3.2 Components Returning Different Receipts

In the usual mPlane communication, the component returns a receipt to the client. This receiptcontains the same parameters and values as the specification, and thus also has the sametoken. The client uses this token to find out which receipt is for which specification. This posesa problem for the service described in section 3.1, because it returns a receipt B that is notbased on the received specification A. Rather, it returns a receipt B for a completely differentservice that handles specification B, and thus the tokens between the original specification Aand the receipt B differ. The client receiving this receipt B then interprets this as a receipt forsomething completely different, even though it was intended as a response to specification A.Because communication uses the HTTP protocol, resolving this turned out to be surprisinglysimple. The HTTP message from the component to the client was extended with a new headermplane-token, which contains the token this message is a response for. So in this example,the header would contain the token for specification A. The client can then know that the receiptB is in fact a response to specification A, even though the tokens are completely different.To make operation smoother in this case, the client automatically invokes a new specificationbased on the receipt, if the token of the receipt and the token in the HTTP header differ. Turningthe receipt into a redemption and sending that to a component wouldn’t work, because it neverreceived a specification and thus never started working and doesn’t know what to do with theredemption. The client turning the receipt into a new specification is completely transparent tothe user and might be surprising at first, because you never receive a result for the exact spec-ification you sent, as is usually the case. The receipt is also removed automatically, preventingdangling receipts that never get removed, but might also be confusing.Another way of handling this would be the component returning a new specification that can beinvoked, removing the need to turn the receipt into a specification on the client’s side. This goesagainst the usual protocol operation though, as specification usually only go from the client tothe component and not the opposite way. Another problem might also appear when the clientalso acts as a component, which is an intended use case. It might then interpret the specifica-tion not as an answer for the client part, but as a message for the component part. For thesereasons, the approach with the receipt was taken.

Chapter 4

Evaluation

As mentioned in chapter 1, there is a problem in the internet with middleboxes that may changeor inhibit traffic and whose locations are not easily found. With the mPlane service describedin sections 3.1 and 3.2, there is now a convenient way to make a large amount of traceroutemeasurements to a single point and get more knowledge about the network topology in thatregion. In this chapter, the mPlane service is used to obtain traceroute measurements frommany to a single point to gain more insight in how the networks looks like and help the locationof failures. Section 4.1 introduces a tool that was written to visualize that traceroute data, andsection 4.2 shows an example how such a visualization might look like and what you can learnjust by looking at it briefly.

4.1 Visualization Tool

To interpret the traceroute results and make them more approachable for humans, a tool waswritten to visualize all traceroutes as a directed graph, where hops are nodes and two neigh-bouring hops in a traceroute have an edge in between. The tool was written in Python usingNetworkX[5] for graph data structures, which uses Graphviz[6] to generate the graph layout.The tool assumes that all traceroutes have the same destination.The tool takes a file containing a mPlane result from the RIPE Atlas service as input, but theactual drawing part is all done with a list of traceroutes, so using different data sources is pos-sible. The traceroutes are visualized on an IP level and on an AS level, using RIPEstat1 to getan AS number for an IP address. The edges and nodes in the graph have different colours andappearances to highlight different properties:

• The red node is the destination of the traceroutes. Optionally, other nodes can be colouredred through a commandline argument.

• Blue nodes signify the starting point of a traceroute measurement.

• All other nodes are coloured according to the AS they belong to. These colours are ran-domized. A legend shows which colour corresponds to which AS. Note that traceroutescan potentially return internal IP addresses and not global ones, and can thus not beattributed to any ASes. In that case, the hop will be coloured as an unknown AS.

• Black edges signify a direct connection between the two hops, i.e. these two hops followedeach other inside a traceroute. An arrowhead shows in which direction the packets flow.

• Red edges in the IP graph signify that there are one or more unknown hops in between,which is the case when e.g. a hop did not respond to a packet. There can also be rededges in the AS graph, but there they have a slightly different and not-so-useful meaning.Red edges there just mean that there is an unknown AS in between (see two pointsabove).

To make the visualization more digestible, a few simplifications have been made:

1https://stat.ripe.net

14

4.2 Example 15

(a) IP Graph (b) AS graph. The positions of the origin ASes donot necessarily correspond to the positions of theorigin IPs

Figure 4.1: Example graphs produced by the visualization tool. Produced by five RIPE Atlasprobes in Germany doing traceroutes to www.spiegel.de (62.138.116.3). Note that theimage format is scalable vector graphics (svg) to support zooming in as close as desired withoutlosing quality

• Paths of nodes that have only one incoming and one outgoing edge have been removedand replaced by a dashed line. The line has a label displaying the ASes the removednodes belonged to. This is only in the IP based graph, as the AS graph is usually alreadymuch simpler.

• If one edge is of particular interest, a commandline switch can highlight every trace thatgoes over that edge. This is done by greying out all other nodes and edges.

• To reduce noise in the graph, the IP address of a hop is selected by majority. Usually,multiple packets get sent per hop, and depending on the traceroute method, those packetscan yield different responses. The tool chooses a valid IP address that appears the mostin those responses, and only considers a hop not responding if all responses are invalid.In a test using paris traceroute with 36,282 hops and three packets each, 96.6% of thehops had the same response for all their packets, so this special consideration is onlyrarely required and the potential error should not be very big.

See appendix B for how to invoke the visualization tool. Figure 4.1 shows a simple example ofhow these graphs could look. Note that all figures shown in this chapter have different figuresizes and legend positions than hardcoded in the tool to fit on the pages, but everything else isunchanged.

4.2 Example

To show a bigger example than in figure 4.1, traceroute measurements were made to www.lottehotel.com at 124.243.25.22, a target that showed path-dependent ECN connectionfailures in the past, located in South Korea. The measurement used 25 probes over eight hours,making a traceroute measurement every five minutes and living up to the name of large-scalemeasurement. This resulted in 283 unique traceroutes, and the visualizations can be seen infigures 4.2 and 4.3.

4.2 Example 16

Figure 4.2: Graph of IP hops to www.lottehotel.com (124.243.25.22)

4.2 Example 17

Figure 4.3: Graph of AS hops to www.lottehotel.com (124.243.25.22)

4.2 Example 18

One thing that one sees immediately is that there are two big ASes handling a lot of the trafficand doing load balancing, which is nicely observable thanks to paris traceroute. These two areLG Dacom (3786) and Korea Telecom (4766), two of the bigger ISPs in South Korea. Another in-teresting observation can be made when looking at the destination; All incoming edges are red,implying that there is something very close to the server that could be tampering with the traffic.Another interesting thing can be observed at the exit of Korea Telecom. While most connectionsexit over the IP address 218.145.43.14 and go to 124.243.2.14 or 124.243.2.18, thereare some red edges coming from inside the AS to 124.243.2.14. This could just be becausethe hop 218.145.43.14 did not respond fast enough, or there could be e.g. a middlebox thatinspects the traffic every once in a while. Armed with those observations, more targeted mea-surements can be made to get more information about a particular point in the network, maybeenhanced with some more specialized measurements other than plain traceroute. This way, it’spossible to close in on a location where a middlebox interferes with traffic.Looking at the graphs, it becomes clear that there are plenty of improvements to be made. Thelayout is entirely computer generated, and that’s noticeable by some questionable node place-ment and edges that go from one end of the graph to the other. Almost by chance, the sourcenodes are more on the top, while the target is towards the bottom. This is probably becausethose nodes have only either outgoing or incoming edges but not both, and Graphviz picks upon that. With a bit of manual node placement, much better results could be achieved.

Chapter 5

Summary and Outlook

In this thesis, two mPlane services to access RIPE Atlas have been presented. They sup-port creating measurements and retrieving results for ping and traceroute measurements. Thisway, specialized connectivity tests made with mPlane can be supplemented by the large-scalegeneric measurements of RIPE Atlas, without having to deal with different interfaces or APIs.Helping the cause of searching for middleboxes that might interfere with connections, a tool hasthen been created that visualizes traceroutes from many points to one target, to get more insightinto how the network is built. The tool was used to show an example of what kind of insight youcan gain by looking at the graphs without much effort, using the mPlane services for RIPE Atlasto gather the data.There is still a lot of work left to do however. The visualization could be improved by man-ual placement of a few key nodes and other placement hints. There is also much potential inautomation here; The manual observations made could also be made by the computer andcoupled with automatic measurements for more detailed reports. The mPlane services for RIPEAtlas are working, but depending on the usage they have to be extended with more parametersor new services have to be created for measurements other than ping and traceroute.Concerning internet measurements in general, it is a bit limiting that most big infrastructures likeRIPE Atlas only offer the basic, well known measurements like ping and traceroute. Approacheslike mPlane are better in customizability, but lack the ability to distribute measurement capabil-ities easily to a large number of probes. It would be very useful to have infrastructures similarto testbeds for wireless sensor networks (e.g. TWIST[7] or FlockLab[8]), where users have fullyprogrammable probes at their disposal.

19

Appendix A

Additions to mPlane

A.1 New Registry Entries

Name Primitive Description

ripeatlas.msm_id natural Measurement ID of a RIPE Atlas measurement

ripeatlas.paris natural Value to control paris traceroute. 0 means standardtraceroute.

ripeatlas.description string The description of a measurement. Auto-generatedif empty.

ripeatlas.probe_id natural Probe ID of a RIPE Atlas probe

ripeatlas.traceroute_id natural A unique identifier for a traceroute to match interme-diate IPs to a traceroute

ripeatlas.hop_index natural Indicates the index of a hop in a traceroute

ripeatlas.paris_id natural Identifies a flow in a paris traceroute measurement

ripeatlas.protocol string Specifies the protocol of a measurement. One ofICMP, TCP or UDP

ripeatlas.probe_source string A string defining a set of probes to be used for a newRIPE Atlas measurement

Table A.1: A summary of all new entries for the mPlane registry of parameters

20

A.2 New Capabilities 21

A.2 New Capabilities

A.2.1 Creating Measurements

Capability ripeatlas-ping-create

Element Constraint

Parameters

when now ... future / 1m

destination.ip4 ∗ripeatlas.probe_source ∗ripeatlas.description ∗ripeatlas.msm_id [∗]

ripeatlas.probe_id [∗]

Results

ripeatlas.msm_id

Table A.2: The ripeatlas-ping-create capability

Capability ripeatlas-trace-create

Element Constraint

Parameters

when now ... future / 1m

ripeatlas.paris 0 ... 64

ripeatlas.msm_id [∗]

ripeatlas.protocol UDP,ICMP,TCP

ripeatlas.probe_id [∗]

ripeatlas.probe_source ∗destination.ip4 ∗ripeatlas.description ∗Results

ripeatlas.msm_id

Table A.3: The ripeatlas-trace-create capability

A.2 New Capabilities 22

A.2.2 Getting Measurement Results

Capability ripeatlas-ping-result

Element Constraint

Parameters

when past ... future

ripeatlas.msm_id ∗Results

delay.twoway.icmp.us.min

delay.twoway.icmp.us.mean

delay.twoway.icmp.us.50pct

delay.twoway.icmp.us.max

delay.twoway.icmp.count

source.ip4

destination.ip4

ripeatlas.probe_id

time

Table A.4: The ripeatlas-ping-result capability

Capability ripeatlas-trace-result

Element Constraint

Parameters

when past ... future

ripeatlas.msm_id ∗Results

intermediate.ip4

ripeatlas.traceroute_id

ripeatlas.hop_index

ripeatlas.paris_id

ripeatlas.protocol

hops.ip

rtt.ms

source.ip4

destination.ip4

ripeatlas.probe_id

time

Table A.5: The ripeatlas-trace-result capability

A.3 New mpcli commands 23

A.3 New mpcli commands

Command Description Usage

savespec Save capabilities and parameters as specificationsin an envelope to a file. If no valid parameters aredefined, they will be prompted for. If no labels or to-kens are given, all capabilities get saved.

savespec [filepath] ([label-or-token] ...)

loadspec Loads a specification from a file and sets all pa-rameters to the found values. All parameters except"when" will be loaded. The specification needs to beinside an envelope.

loadspec [filepath] [label]

loadrunspec Loads a specification from a file, runs it, and sets allparameters to the found values. All parameters ex-cept "when" will be loaded. The specification needsto be inside an envelope.

loadrunspec [filepath] [label]

savemeas Save results of measurements in an envelope to afile. If no labels or tokens are given, all results getsaved.

savemeas [filepath] ([label-or-token] ...)

Table A.6: New commands available for mpcli

Appendix B

Visualization Tool Usage

The tool requires Python 3 and the following packages:

• mPlane

• NetworkX

• PyGraphviz (and a system installation of Graphviz itself)

• Matplotlib

The working directory needs to be the folder with the script and all other files.

Usage: visualization.py [filepath] [dest] (-red [addr]...)(-o [outdir]) (-highlight [addr] [addr])

Argument Description

filepath Path to a file containing a mPlane result. If the file contains a mPlane envelope, the first element will be taken.

dest The destination of the traceroutes.

-red (Optional) IP addresses that should be coloured red. The ASes that contain these addresses will be colouredred in the AS graph as well.

-o (Optional) Output directory for the pictures.

-highlight (Optional) If these two addresses are specified, all traceroutes that go through those two hops will be high-lighted by greying out all other traces. Same for the ASes in the AS graph.

Table B.1: Usage for the visualization tool (visualization.py)

24

List of Abbrevations

AS . . . . . . . . . . . . . . . Autonomous System, an entity that operates a network within the internetand routes internet traffic through it.

ASN . . . . . . . . . . . . . . Autonomous System Number, a number uniquely identifying an au-tonomous system.

ECN . . . . . . . . . . . . . . Explicit Congestion Notification, an extension to TCP and IP to allow sig-nalling of congestion without initial packet loss.

HTTP . . . . . . . . . . . . . HyperText Transfer Protocol, a stateless protocol to transmit data basedon requests and responses.

ICMP . . . . . . . . . . . . . Internet Control Message Protocol, a protocol to exchange administrativemessages.

IP(v4/v6) . . . . . . . . . Internet Protocol, the communication protocol for the internet, version fouror six.

JSON . . . . . . . . . . . . JavaScript Object Notation, a human-readable way of storing attribute andvalue pairs.

REST . . . . . . . . . . . . REpresentational State Transfer, an architecture for resource access fol-lowing certain guidelines.

RIPE . . . . . . . . . . . . . Réseaux IP Européens, the European internet registry.RTT . . . . . . . . . . . . . . Round Trip Time, the time it takes for a packet to travel to a destination and

back.TCP . . . . . . . . . . . . . . Transmission Control Protocol, a protocol for reliable transport of data.TTL . . . . . . . . . . . . . . Time To Live, a counter in a packet. If it reaches zero, the packet gets

dropped.UDP . . . . . . . . . . . . . . User Datagram Protocol, a protocol for unreliable transport of data.URL . . . . . . . . . . . . . . Uniform Resource Locator, specifying a location where and the means how

to retrieve a resource.

26

List of Figures

2.1 Structure of mPlane and RIPE Atlas . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Messages exchanged between client and component in mPlane . . . . . . . . . 6

3.1 Depiction of how traceroute lists get unrolled . . . . . . . . . . . . . . . . . . . . 12

4.1 Example graphs produced by the visualization tool . . . . . . . . . . . . . . . . . 154.2 Graph of IP hops to www.lottehotel.com (124.243.25.22) . . . . . . . . . 164.3 Graph of AS hops to www.lottehotel.com (124.243.25.22) . . . . . . . . . 17

27

List of Tables

3.1 Assignment of RIPE Atlas parameters to mPlane parameters when creating mea-surements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.2 String tokens when requesting RIPE Atlas probes summarized . . . . . . . . . . 103.3 Result columns and their types in a ping result . . . . . . . . . . . . . . . . . . . 103.4 Result columns and their types in a traceroute result . . . . . . . . . . . . . . . . 11

A.1 A summary of all new entries for the mPlane registry of parameters . . . . . . . . 20A.2 The ripeatlas-ping-create capability . . . . . . . . . . . . . . . . . . . . . 21A.3 The ripeatlas-trace-create capability . . . . . . . . . . . . . . . . . . . . 21A.4 The ripeatlas-ping-result capability . . . . . . . . . . . . . . . . . . . . . 22A.5 The ripeatlas-trace-result capability . . . . . . . . . . . . . . . . . . . . 22A.6 New commands available for mpcli . . . . . . . . . . . . . . . . . . . . . . . . . 23

B.1 Usage for the visualization tool (visualization.py) . . . . . . . . . . . . . . . . . . 24

28

Bibliography

[1] B. Trammell, M. Kühlewind, D. Boppart, I. Learmonth, G. Fairhurst, and R. Scheffenegger,“Enabling internet-wide deployment of explicit congestion notification”, English, in Passiveand Active Measurement, ser. Lecture Notes in Computer Science, J. Mirkovic and Y. Liu,Eds., vol. 8995, Springer International Publishing, 2015, pp. 193–205, ISBN: 978-3-319-15508-1. DOI: 10.1007/978-3-319-15509-8_15. [Online]. Available: http://dx.doi.org/10.1007/978-3-319-15509-8_15.

[2] B. Trammell, P. Casas, D. Rossi, A. Bar, Z. Houidi, I. Leontiadis, T. Szemethy, and M. Mellia,“Mplane: An intelligent measurement plane for the internet”, Communications Magazine,IEEE, vol. 52, no. 5, pp. 148–156, 2014.

[3] N. RIPE. (2010). Ripe atlas, [Online]. Available: https://atlas.ripe.net.

[4] B. Augustin, X. Cuvellier, B. Orgogozo, F. Viger, T. Friedman, M. Latapy, C. Magnien, andR. Teixeira, “Avoiding traceroute anomalies with paris traceroute”, in Proceedings of the6th ACM SIGCOMM Conference on Internet Measurement, ser. IMC ’06, Rio de Janer-iro, Brazil: ACM, 2006, pp. 153–158, ISBN: 1-59593-561-4. DOI: 10.1145/1177080.1177100. [Online]. Available: http://doi.acm.org/10.1145/1177080.1177100.

[5] A. Hagberg, P. Swart, and D. S Chult, “Exploring network structure, dynamics, and functionusing networkx”, in. Jan. 2008. [Online]. Available: http://www.osti.gov/scitech/servlets/purl/960616.

[6] J. Ellson, E. Gansner, L. Koutsofios, S. North, and G. Woodhull, “Graphviz— open sourcegraph drawing tools”, English, in Graph Drawing, ser. Lecture Notes in Computer Science,P. Mutzel, M. Jünger, and S. Leipert, Eds., vol. 2265, Springer Berlin Heidelberg, 2002,pp. 483–484, ISBN: 978-3-540-43309-5. DOI: 10.1007/3-540-45848-4_57. [Online].Available: http://dx.doi.org/10.1007/3-540-45848-4_57.

[7] V. Handziski, A. Köpke, A. Willig, and A. Wolisz, “Twist: A scalable and reconfigurabletestbed for wireless indoor experiments with sensor networks”, in Proceedings of the 2NdInternational Workshop on Multi-hop Ad Hoc Networks: From Theory to Reality, ser. REAL-MAN ’06, Florence, Italy: ACM, 2006, pp. 63–70, ISBN: 1-59593-360-3. DOI: 10.1145/1132983.1132995. [Online]. Available: http://doi.acm.org/10.1145/1132983.1132995.

[8] R. Lim, F. Ferrari, M. Zimmerling, C. Walser, P. Sommer, and J. Beutel, “Flocklab: A testbedfor distributed, synchronized tracing and profiling of wireless embedded systems”, in Infor-mation Processing in Sensor Networks (IPSN), 2013 ACM/IEEE International Conferenceon, Apr. 2013, pp. 153–165. DOI: 10.1109/IPSN.2013.6917582.

29