Packet Injection

8/3/2019 Packet Injection

1/17

Di P Ijia guIDe to observIng Packet sPooIng by IsP

b sh sh

h@.

eLectronIc rontIer ounDatIon.


2/17eLectronIc rontIer ounDatIon e.org

vi .0 nm 28, 2007

Detecting Packet Injection:A Guide To Observing Packet Spoofng by ISPs

Idi

Certain Internet service providers have begun to interere with their users communications byinjecting orged or spooed packets data that appears to come rom the other end but wasactually generated by an Internet service provider (ISP) in the middle. Tis spoong is onemeans (although not the only means) o blocking, jamming, or degrading users ability to useparticular applications, services, or protocols. One important means o holding ISPs account-able or this intererence is the ability o some subscribers to detect and document it reliably.We have to learn what ISPs are doing beore we can try to do something about it. Internetusers can oten detect intererence by comparing data sent at one end with data received at theother end o a connection.

echniques like these were used by EFF and the Associated Press to produce clear evidence

that Comcast was deliberately interering with le sharing applications; they have also beenused to document censorship by the Great Firewall o China.1 In each o these cases, an in-termediary was caught injecting CP reset packets that caused a communication to hang up even though the communicating parties actually wanted to continue talking to one another.In this document, we describe how to use a network analyzer like Wireshark to run an experi-ment with a riend and detect behavior like this. Please note that these instructions areintended or use by technically experienced individuals who are generally amiliar withInternet concepts and are comortable installing sotware, examining and modiying theircomputers administrative settings, and running programs on a command line.

rqim

Making use o these techniques requires some general understanding o Internet technologyand some technical expertise. I you dont understand the process, you may not produce mean-ingul evidence about what your ISP is doing. Although we have attempted to explain mosto the network concepts and principles involved, it may prove helpul to have read at least onetechnical book or web site about the CP/IP protocol suite beore beginning.

Te test described here must be perormed in conjunction with a riend who is using a dier-ent Internet connection (and thereore is probably in a dierent location). Both you and yourriend must have a good understanding o the process described here; this test relies on com-paring observations made at two dierent locations in order to nd dierences between them,

so it would not be meaningul i perormed by one party alone. Tereore, these instructionsare primarily useul or testing peer-to-peer applications or applications or which you can run

1 See Clayton, Murdoch, and Watson, Ignoring the Great Firewall o China (available at http://www.cl.cam.ac.uk/~rnc1/ignoring.pd).

2 I your ISP were blocking Google, you might suspect this when you encountered diculty connecting toGoogles services and you might see apparently anomalous packets in a packet trace rom your end. However,only the addition o a recording o the same interactionfrom Googles end would denitively establish whetherthe problem lay with Google or with an ISP in between. Otherwise, it is dicult to tell whether the problemis a result o ordinary packet loss, a misbehaving computer at Googles end, or even misbehaving sotware onyour own computer.
http://www.cl.cam.ac.uk/~rnc1/ignoring.pdfhttp://www.cl.cam.ac.uk/~rnc1/ignoring.pdfhttp://www.cl.cam.ac.uk/~rnc1/ignoring.pdfhttp://www.cl.cam.ac.uk/~rnc1/ignoring.pdf



your own server. It is thereore dicult to conrm i an ISP is blocking a third-party servicelike Google unless the operator o that service is interested in participating directly in the tests.2

Te tests described here are most relevant as a means o debugging a specic observed andreproducible problem (or example, an inability to connect to another party) rather than as aspeculative means o investigating ISP behavior. Tis is primarily because o the limitations otools to automate the process o comparing packet traces rom two ends o a connection. ra-ditionally, this comparison had to be perormed by hand, which can be a quite laborious pro-

cess i one isnt looking or anything in particular. EFF has begun to develop tools to automatethis process so that large packet traces can be compared automatically and packet injection canbe detected even when it is not specically suspected.

Each party participating in the experiment must have all o the ollowing:

a computer capable o running Wireshark, with appropriate privileges to install andrun it;

the ability to connect this computer directly to the Internet, with apublic IP address,outside o any frewalls (or example, not via a typical home wireless router);

the ability to determine the computers public IP address;

the ability to disable any frewall sotware running on the computer itsel;

some application to test, and the ability to congure that application to communicatedirectly with the other party (by IP address).

a pi

Using a packet snier can capture all o the trac passing by your computer (including all oyour communications, and potentially communications o other users on the same network);i your computer is connected to a wireless network, or example, the packet snier may recordeverything you do and everything everyone else on the wireless network does on-line. Pleasedo not record other peoples communications without their consent. Doing so is impolite and,under some circumstances, may be prohibited by law. One way to avoid recording third parties

We urge readers to interpret the results o their tests cautiously and avoid jumping to conclusions or makingspurious accusations. For example, RS packets are a legitimate part o the CP protocol, and receiving RSpackets does not normally mean that they were spooed by an intermediary. RS packet spoong can onlybe proven denitively by making simultaneous measurements at the endpoints o a connection. (Some ormso packet spoong could produce suggestive evidence at one end because the spooed packets have anomalousproperties that make it very unlikely that they were really transmitted by the other end. But observing theseanomalies will probably not be truly conclusive unless evidence is also gathered at the other end.)

It would clearly be useul to have a reliable automated means o testing non-P2P services to detect intererenceor degradation by ISPs. In some cases this would just be a matter o writing sotware, while in other cases itwould require co-ordination among multiple parties (such as Google in the example just given). For example, itwould be helpul to know whether ISPs give users lower-latency connections to some web sites than to others.Because the web site operators co-operation is required or this test, it is beyond the scope o this article. Onetest that could be perormed readily by two end-users is seeing whether exchanging the same volume of data withdierent protocols (or example, sending a single 1-megabyte le with HP, FP, Bitorrent, Gnutella, anddisguised as a SIP VoIP telephone call data stream) takes appreciably dierent amounts o time, and whether theround-trip latency or each o these protocols is the same or not. Tis article does not discuss tools that wouldhelp automate such a test. We do discuss our pcapdi tool below and also mention the University o Washing-ton research project using JavaScript to detect some modication o web page contents by ISPs.



communications is to avoid usingpromiscuous mode or your packet capture, unless you speci-cally need it.

I you produce a capture le (packet trace) with evidence o the results o your experiment,please be aware that the capture le will reveal your IP address, the IP address o the other per-son involved, and a complete record o everything you did on-line during the course o the ex-periment. For example, i you downloaded a le, the packet trace will typically reveal which le(and even include the ull contents o that le). I you browsed the web or checked your e-mail

while the packet snier was running, the identities and contents o web pages you visited and e-mail messages you downloaded may appear in the packet trace. In addition, any HP cook-ies sent by your browser (which might include your username and password or web sites youvisited!) will be included in the trace. You should exercise caution when publishing or sharing apacket capture le to ensure that you dont reveal more inormation than you intended.

th

Te traditional Internet architecture is characterized by an end-to-end design in which ISPspassively orward unmodied packets rom one user to another. Tis means that, in the bestcase, every packet sent by one user should be received as an identical packet at the other end.

Tere are several reasons that this ideal might not be attained even in the absence o packetinjection by ISPs:

mi Te Internet Protocol standard per-mits ISPs toragment packets that are too large (orexample, because a particular network technologyused by an ISP has a maximum packet size). Tepackets are then broken up into smaller ragmentswhich arrive separately at the destination; the desti-nation computer is responsible or reassembling theragments. Fragmentation has become somewhat

less common in practice or reasons that may includeconservative packet size deaults in operating sys-tem network code and mechanisms like path MUdiscovery (to automatically select a packet size thatis small enough to avoid ragmentation). In test results weve seen so ar, ragmentationgenerally did not occur, and we will ignore this possibility here, although it should beconsidered as a possible cause o any observed discrepancies between packets sent andreceived.

P l. Under conditions o network conges-tion, it is normal or some packets to be discarded

rather than orwarded, a phenomenon calledpacketloss. Packet loss is normally measured as a percent-age; the ping utility measures packet loss with ICMPecho request packets, counting how many ICMP echoreplies are received in response to a certain numbero probes. High rates o packet loss could be causedintentionally by an ISP as a means o reducing theperormance o a targeted application or protocol,but they can also occur as a result o congestion on

ISPNetworks

ISPNetworks



the network or other technical problems. When a packet is lost (also called a droppedpacket, dropped rame, or dropped segment), it is not received by the destination atall. Some higher-level Internet protocols include mechanisms or coping with packetloss, such as CPs mechanism or explicitly retransmitting data rom packets that arelost.

rdi. Sometimes packets are not deliveredin the same order in which they were originally

transmitted. I packet B was transmitted aterpacket A, receiving packet B at the other enddoes not mean that packet A has been dropped;it might still be on its way. CP can also gener-ally correct or reordering. Like packet loss, reor-dering could be used intentionally by an ISP todegrade an application, but also occurs normallyin the course o Internet routing. Tereore, ob-serving reordering does not necessarily indicate a

problem or anomaly.

spf. Spoong orpacket injection occurswhen an entity other than one o the endpointsgenerates trac using the source address o anendpoint. Spoong is the most straightor-wardly detectable means o intererence withInternet trac because it produces concreteevidence in the orm o the anomalous spooedpackets, and because it does not occur normallyin traditional Internet routing.

Spoong can be detected by looking or packets that

were received by one end but never sent by the otherend. I user B receives a packet apparently rom user Athat user A has no record o having sent, user B can conclude that someone in between the twohas spooed this packet. Te remainder o this article describes the means o collecting packettraces to allow the packets actually transmitted between two users to be compared in this way.

Spoong need not involvepreventingor blockingcommunications; it could also involve changingtheir content, as with a transparent proxy.3 Some ISPs have been experimenting with modiy-ing HML in third-party web pages on the y in order to inject advertising. Here is a recentreal-world example:

ISPNetworks

ISPNetworks

3 One Internet user congured his open wireless gateway to modiy images seen by users as they browsed theweb, either by mirror-reversing them or blurring them. See http://www.ex-parrot.com/~pete/upside-down-ternet.html. It can be easy to orget that Internet intermediaries are able to make arbitrary changes to theirusers view o the Internet.
http://www.ex-parrot.com/~pete/upside-down-ternet.htmlhttp://www.ex-parrot.com/~pete/upside-down-ternet.htmlhttp://www.ex-parrot.com/~pete/upside-down-ternet.htmlhttp://www.ex-parrot.com/~pete/upside-down-ternet.html



In this image supplied by Chris Palmer, a wireless ISP has modied the source code o Googleshome page to add its own advertising, presumably by spoong packets with a transparentHP proxy. Although similar behavior oten results rom adware, in this instance Palmerveried that the ISP was at ault by installing a resh instance o Windows inside a virtualmachine. 4

sp

sp . Install Wireshark

Download a copy o Wireshark or your platorm rom the Wireshark home page at http://www.wireshark.org/. (Wireshark is also prepackaged or most Unix-like operating systemsand may be available rom your distributors package repository. In older operating systemreleases, it may still be packaged under its ormer name, Ethereal.) Install Wireshark and makesure that you can run the program. 5

4 Tis particular advertising appears to have been added by AnchorFree, one o several rms experimenting withthis means o ad placement; see http://anchorree.com/advertisers-agencies/how-it-works/. See also http://vancouver.cs.washington.edu/or a research project investigating the prevalence o this phenomenon.

5 I youre running Wireshark on the same machine thats generating or receiving the test trac, as werecommend here, you dont need to ollow the additional directions at http://www.wireshark.org/aq.html#promiscsnibecause you wont need to capture trac in promiscuous mode. Running Wireshark inpromiscuous mode on a dierent machine on the same local area network segment (note: not on an Ethernetswitch) could, however, help mitigate problems with excessive CPU load, with the unavailability o Wiresharkor a suitable packet capture driver or a particular operating system or device, with CP or UDP checksumofoading or large segment ofoading (described below), or when logging into a remote server by means suchas SSH in order to run tests on that server s communications with your client machine. In this scenario, the
http://www.wireshark.org/http://www.wireshark.org/http://anchorfree.com/advertisers-agencies/how-it-works/http://vancouver.cs.washington.edu/http://vancouver.cs.washington.edu/http://www.wireshark.org/faq.html#promiscsniffhttp://www.wireshark.org/faq.html#promiscsniffhttp://www.wireshark.org/faq.html#promiscsniffhttp://www.wireshark.org/faq.html#promiscsniffhttp://vancouver.cs.washington.edu/http://vancouver.cs.washington.edu/http://anchorfree.com/advertisers-agencies/how-it-works/http://www.wireshark.org/http://www.wireshark.org/



sp 2. Connect directly to the Internet

In order to obtain the most valid and conclusive results, we strongly recommend that your com-puter be directly connected to the Internet, with a globally-valid public IP address, without anyrewalls or network address translation (NA) routers. Perorming these tests rom behindNA could produce valid results but creates some uncertainty about whether unexpectednetwork behavior is due to an ISP or a local NA router. A public IP address is one that an-other party can use directly to communicate with you without the need to congure a tunnel or

rewall rule or use a proxy to connect. 6

I you are using an institutional network connection, such as at a school or business, that has arewall that you are not permitted to disable, you may still be able to perorm these tests, butany packet spoong you detect may be a result o your institutions rewall rather than its up-stream ISP connection. As we describe below, observing packet spoong shows that someoneis doing it, but does not directly reveal who. We want to reduce the number o possible respon-sible parties.

Tereore, beore beginning the test, disable any rewalls and NAs located between yourcomputer and the Internet including both sotware or personal rewalls running on yourcomputer and rewall appliances or rewall unctionality in routers. As an alternative, connectyour computer to a point on the network located outside o any rewalls or NA routers (orexample, by directly connecting it to a cable modem or DSL modem). Some network designsand documentation reer to this location as a DMZ.

sp . I possible, disable CP and UDP checksum ooading and CP segmentation ooading

Checksum ofoading (sometimes called CP checksum ofoading, although UDP check-sums may also be ofoaded) is a eature o some recent Ethernet cards, particularly GigabitEthernet-capable cards, that allows the Ethernet card to construct portions o some networkpackets in hardware, saving load on the CPU. However, the use o checksum ofoading makespacket captures inaccurate because it prevents the local operating system rom seeing what was

actually transmitted. Tis may cause a discrepancy since one end mistakenly thinks it sentsomething slightly dierent rom what the other end correctly received; the resulting mismatcho CP or UDP checksum values could be misinterpreted as tampering by an ISP, since ISPsare not supposed to alter these checksums. Checksum ofoading should, i possible, be dis-abled at both ends beore beginning the experiment. I you know that your Ethernet card ornetwork driver does not perorm checksum ofoading, you do not need to disable it. It mayalso be possible to get valid results when checksum ofoading is enabled; workarounds or this

machine capturing packets is not the same machine that generates them; however, the resulting packet trace cangenerally be used in the same way as a packet trace that was captured directly on the machine generating thepackets, as long as the LAN to which the machines are connected broadcasts all packets to the packet-captur-ing machine. Tere are also techniques or sning trac on some non-broadcast switched LANs, which arebeyond the scope o this document.Running Wireshark on MacOS X requires X11; there are other pcap-compatible native packet sniers orMacOS X.

6 NA devices and rewalls create uncertainty because they routinely rewrite, drop, or block trac; i they arenot disabled, it will be dicult to prove that communications that blocked or altered packets were blockedor altered by an ISP rather than by a rewall device. NAs and proxies also prevent direct packet-by-packetcomparison o packet traces because the end points do not have a consistent view o the source and destinationaddresses in use, and there may not even be a one-to-one relationship between packets entering and exiting aNA or proxy. Evidence o third-party packet tampering gathered in the presence o any o these devices isbetter than no evidence, but must be interpreted with extreme caution.



purpose are described in a later section. Here are typical means o disabling checksum ofoad-ing on several popular operating systems:

On Linux (as root):

ethtool -K eth0 rx o tx o (choose correct network interace i not eth0)

On FreeBSD (as root):

iconfg em0 -rcxsum -tcxsum (choose correct network interace i not em0)

On MacOS (as root):

sysctl -w net.link.ether.inet.apple_hwcksum_tx=0

sysctl -w net.link.ether.inet.apple_hwcksum_rx=0

(Note that this may cause some local applications to work incorrectly!)

On Windows: right-click My Computer, then select Device Manager / Network Adapters /(select device) / Properties / Advanced; then disable checksum ofoading, i the option is avail-able.

For general inormation about checksum ofoading and why it can cause errors when captur-ing packets, see http://www.wireshark.org/docs/wsug_html_chunked/ChAdvChecksums .html and http://www.wireshark.org/aq.html#q11.1. Note that the approach suggested thereo disabling CP checksum verication in Wireshark does not help or our purposes, becausewe want to compare packets; having CP checksums that are dierent across capture les willstill appear as a discrepancy between those capture les even i the checksums values are neververied.

I your system perorms checksum ofoading and you are unable to disable it, other options areavailable. Te pcapdi program described below allows you to ignore CP and UDP check-sum values entirely, in case you have reason to believe that checksum ofoading is in use. You

can also perorm the capture on a separate machine distinct rom the computer that is generat-ing the test trac as long as it is connected to the same local area network and is able to seethe trac passing by. In this case, the capture should be perormed in promiscuous mode (seeootnote 3 above) and, on a network used by multiple people, extra care should be taken toavoid capturing other users communications without their knowledge.

Another orm o ofoading that can cause packet capture accuracy problems is CP segmenta-tion ooading, also known as large segment ooad. In CP segmentation ofoading, an Ether-net card, rather than operating system sotware, splits a large CP packet into multiple CPpackets. Tis can cause a serious discrepancy in the number o packets a host believes it trans-mitted as against the number o packets it actually transmitted; since packets are split up by the

network card in a way invisible to the senders operating system, every CP packet large enoughto be split may appear to be orged, since the sender will have no record o having sent anyo the received packets in the orm in which they were received. CP segmentation ofoad-ing should also be disabled i your system uses it. pcapdi, or example, is not able to ignoreCP segmentation discrepancies in the same way that it can ignore CP and UDP checksummismatches. See http://www.inliniac.net/blog/2007/04/20/snort_inline and tcp segmenta-tion ofoading.html or a discussion o CP segmentation ofoadings consequences or packetsning, and inormation about disabling it on Linux. In general, the validity o the results opacket capture experiments will be improved by disabling all available ofoading eatures.
http://www.wireshark.org/docs/wsug_html_chunked/ChAdvChecksums.htmlhttp://www.wireshark.org/faq.html#q11.1http://www.inliniac.net/blog/2007/04/20/snort_inline%20and%20tcp%20segmentation%20offloading.htmlhttp://www.inliniac.net/blog/2007/04/20/snort_inline%20and%20tcp%20segmentation%20offloading.htmlhttp://www.inliniac.net/blog/2007/04/20/snort_inline%20and%20tcp%20segmentation%20offloading.htmlhttp://www.inliniac.net/blog/2007/04/20/snort_inline%20and%20tcp%20segmentation%20offloading.htmlhttp://www.wireshark.org/faq.html#q11.1http://www.wireshark.org/docs/wsug_html_chunked/ChAdvChecksums.html



sp . Determine local IP address

Next, determine the IP address o your computer. You can obtain this locally rom your com-puters network conguration tools, and you can also obtain it rom a web site such as http://whatismyipaddress.com/ or http://www.whatismyip.com/, which displays the IP address romwhich it is being accessed. Use both methods to ensure that you are really directly connected tothe Internet and not using a proxy server or NA connection. (I you are not using NA, yourcomputers locally-congured IP address should be identical to the IP address seen by web

sites and other Internet users. I they still disagree, its possible that your ISP or institutionalnetwork is orcing all users to use NA or a proxy, rather than providing direct access to theInternet.)

From here on, we will suppose that your IP address is 12.13.14.15 and that the IP address othe person at the other end is 4.8.16.32. You should replace these example addresses with thereal IP addresses involved. 7

sp . Conrm IP addresses and test connectivity

o ensure that both you and the person at the other end have correctly determined your IPaddresses, try to perorm some operation that allows you to communicate with one another

by speciying each others IP addresses. You could use the ping command or try the applica-tion that you eventually plan to test out, such as a P2P le-sharing client or VoIP application.I you cant establish some kind o end-to-end communication using the IP addresses youvedetermined, youll need to debug this problem beore proceeding any urther. Possible causescould include the presence o a rewall, including a sotware rewall, that has not yet beendisabled at one end.

sp . Synchronize computer clocks

I your operating system supports it, make sure that your computers clock is synchronized toan authoritative Network ime Protocol (NP) network time server, so that the dates and

times recorded in your packet capture will be accurate and will correspond to those recorded bythe other computer. Tis will help make results or log entries rom multiple computers easierto compare.

ri h

Start Wireshark (with administrative privileges, e.g. root privileges, sucient to perorm araw packet capture8). Select Interaces rom the Capture menu. Choose the interace cor-responding to the network device you will use to capture packets; the IP addresses bound toall available network interaces are displayed, which may help you distinguish them i you areunsure which network interace is which. Click the Options button or the correct interace.

In the Capture Options dialogue, ensure that the IP address you expected your computer to beusing is displayed in the IP address eld. We recommend setting a capture lter to ensure that onlypackets directly to or rom the other computer will be captured. I the other computers IP ad-

7 Note that i your IP address appears to be in a private address range dened by RFC 1918 (10.0.0.0 10.255.255.255, 172.16.0.0 172.31.255.255, or 192.168.0.0 192.168.255.255), you have not properlyollowed the instructions to disable any rewalls and NA devices (or your ISP is orcing everyone to use itsown ISP-operated NA service).

8 Windows users may also be able to use the NPF driver to perorm a packet capture as an unprivileged user.
http://whatismyipaddress.com/http://whatismyipaddress.com/http://www.whatismyip.com/http://www.whatismyip.com/http://whatismyipaddress.com/http://whatismyipaddress.com/



dress is 4.8.16.32, a suitable capture lterstring is host 4.8.16.32. We alsorecommend setting Update list o packetsin real time and Automatic scrolling in livecapture, which help you watch the captureprocess while its underway, unless yourcomputer is too slow.

When youre ready to begin capturingpackets, click the Start button. I youveset a capture lter to limit capturing totrac to or rom the other computer, youwill probably not see any packets appear inthe capture until you deliberately generatesome trac between the two computers.o ensure that the trac is showing up,you should ensure that Wireshark packetcaptures are running at both ends, andthen have one computer ping the other

computer by IP address. Tis generatesa steady stream o ICMP echo request and echo reply packets. Current Unix, Windows, andMacOS operating systems all allow you to start the ping process by typing ping 4.8.16.32 at aterminal (command-line) prompt. (On some systems, the ping will stop automatically ater apredetermined number o pings; on others, you can interrupt it by pressing Ctrl+C.)

I both ends o the connection are capturing data, the ICMP packets that represent ping re-quests and replies should appear in the Wireshark window at each end. Tese packets shouldbe identical at the IP layer and unless the ping utility itsel reported packet loss thereshould be an identical number o packets seen rom both ends. Tis test will show i the cap-ture process is set up and working properly.



Here, we see how the capture process appears at both ends ollowing a large number o pingrequests rom 12.13.14.15 to 4.8.16.32. Te packets displayed agree one-to-one (as could beveried by looking more closely at the packets contents than we do here); there are no droppedpackets and no spooed packets. (Note that you cant see this view rom both ends in real-time;this screen shot was created ater the act by copying a saved capture le rom one computer

onto another. While the capture is running, each party sees only one o the two windowsdisplayed here; in order to veriy that the packet captures correspond properly while theyrein progress, youll need to use some other means o communication to talk to the person atthe other end, such as a telephone or instant messaging application. Youll use this channel tocoordinate your activities and to compare notes.)

Once youre condent that both computers are talking to each other over the Internet correctlyand are saving a valid packet trace, you can begin to gather experimental data about whateverapplication is o interest to you or try to reproduce any reported or conjectured errors orproblems. For example, or our tests with Comcast, we congured one o the computers as aBitorrent tracker and seeder and gave the other computer a Bitorrent le instructing it to at-

tempt to download the le hosted by the rst computer. Te details o what youll test dependon what youre interested in and will require you to be amiliar with the application youretesting in some detail. (In the Bitorrent example, its not sucient simply to have both partiesstart running Bitorrent at this point; rather, one computer needs to be congured explicitlyto oer a download to the other computer, which, in turn, needs to be congured to request adownload rom that computer.)

When your experiment is complete, you should stop the capture and save the resulting captureles to disk. You can then exchange these les with the person at the other end by e-mail, orinstance. Wireshark can be used to open and display a saved capture le generated on anothercomputer.

Tere are many other network analyzers or packet sniers that could be used instead o Wire-shark. We have chosen to describe Wireshark because it is powerul, user riendly, open source,and available or several platorms. As long as the network analyzer you use can save its packettraces in pcap ormat, they can be read by a wide variety o sotware, including Wireshark; thus,meaningul and comparable results could be obtained using other sotware. I youre capturingpackets on a device that cant run Wireshark such as a remotely-accessible server in a colo-cation acility with no graphical user interace you should consider the tcpdump program,which is a standard part o many Unix-like operating systems and is available at http://www.tcpdump.org/.
http://www.tcpdump.org/http://www.tcpdump.org/http://www.tcpdump.org/http://www.tcpdump.org/



It is important to be aware that some network analyzers deault to capturing only a portion oeach packet. Wiresharks option to Limit each packet to 68 bytes in the screen capture aboveis (correctly) disabled by deault, but the corresponding option in some other programs couldbe (wrongly) enabled by deault. o meaningully compare the resulting packet traces, thepacket size limit should be set to the largest possible value (usually 65535 bytes) or disabledentirely. For example, using the tcpdump utility, an appropriate command line would be

tcpdump -v -s 0 -w packet_trace.pcap

Setting -s 0 sets the packet size limit to unlimited instead o tcpdumps deault o 68 bytes.(When running tcpdump on a computer with multiple network interaces, it may also be nec-essary to speciy a network interace with the -i option.)

Ipi h l

Packet trace comparison

As we described above, packet trace les generated on separate machines that were communi-cating directly with one another can be compared to see how packets sent by each computer

correspond to packets received by the other computer. Since a simple le transer or VoIPconversation could generate thousands o packets, the lack o automated tools to perorm thiscomparison can make the process tedious. We intend to collaborate with other interested par-ties to produce such comparison tools in the near uture, enabling much larger data sets to beanalyzed quickly or spooed packets, even where specic sorts o spoong are not suspected.Here are a ew tips or comparing packet traces by hand:

Spooed packets may correspond to protocol errors or extreme delays reported orobserved in application sotware; or example, i a client program gives an error likeConnection reset by peer, Connection closed by oreign host, Lost connection, etc.,or data rates suddenly drop, packet spoong may have occurred. Noting the timing

o suspicious errors may provide guidance or where to look or spooed packets in apacket trace le.

Spooed packets used to disrupt connections are oten CP segments with the FINor RS ags set (also known as FIN packets and RS packets); each o these agsindicates that a computer does not want to continue a CP conversation. Wiresharkcan be congured to color these packets dierently so that they stand out in a packetdisplay. Keep in mind that there are legitimate uses or FIN and RS packets withinthe CP standards and that the presence o orged FIN or RS packets, not the



presence o such packets generally, is suspicious. (A client sotware or rewall bug, orexample, could cause one end o a connection to disconnect prematurely but that isntthe ISPs ault!)

I a problem youre investigating is widespread, someone may already have publishedclaims about precisely when or under what circumstances spooed packets may appear;you can consult the details o other peoples allegations to see whether you can repro-duce these claims or yoursel.

Setting aside the possibility o ragmentation, we have explained that packets are spooed whenthey are received by one computer but were not transmitted by the other computer. Below, wegive examples o summary views showing a Bitorrent transer disrupted by CP RS spoong:



Beore the RS packets begin (packet no. 2338 in the local capture and packet no. 2549 in theremote capture9), the packets transmitted and received correspond directly to one another (al-though they appear in rather dierent orders). 10 Once the RS packets begin, a large number

o packets are received at each end that do not correspond to packets transmitted at the otherend. We can veriy this by looking at the detailed contents o these packets (or example, theirsequence numbers, which are displayed by Wireshark as Seq=nnnnn), although simply count-ing them tells the tale in this case. Te local machine, with IP address 12.13.14.15, reportedtransmitting a total o ve RS packets to the remote machine at 4.8.16.32 (packet nos. 2340,2342, 2346, 2350, and 2354), while it reported receiving 13 such packets rom 4.8.16.32(packet nos. 2338, 2339, 2343, 2344, 2347, 2348, 2351, 2352, 2355, 2356, 2357, 2358, and2359). Te remote machine, with IP address 4.8.16.32, reported transmitting only a singleRS packet (packet no. 2560) while it believes it received 17 RS packets rom 12.13.14.15(2549, 2550, 2551, 2552, 2553, 2554, 2555, 2556, 2557, 2558, 2561, 2562, 2563, 2564, 2565,

9 In this context, packet number reers to the ordinal number o the packet within a particular capture le,which is displayed by Wireshark in the let-hand column; the rst packet sent or received is number 1, the nextnumber 2, and so on. Tere are also other orms o serial numbering contained within the packets themselves,such as the IP identication eld and CP sequence number. pcapdi uses the IP identication eld (whichWireshark reers to as ip.id or display lter purposes) to reer to packets, but this paragraph is merely discuss-ing Wireshark packet ordinal numbers. Packet no. 2340 in this sense need not have IP identication value2340.

10 Tis reordering is not necessarily only due to ISP packet reordering, but also to the existence o network la-tency; i both computers transmit a packet at noon over a network with a 1-second latency, each computer willreceive the other computers transmissions at 12:00:01 and consider its own transmission to have taken placerst.



2566, 2567). Evidently, many o the RS packets received by each machine did not actuallyoriginate rom the other.11 Te result o these RS packets in this case was that the Bitorrentsession stopped and did not resume or around our and a hal minutes. (A new CP sessionis established with the characteristic SYN and SYN/ACK packets at local capture packet nos.2446 and 2447, which correspond to the remote captures packet nos. 2578 and 2579.)

Wireshark provides the ability to view not only a packet summary list like the lists displayedabove but also the sequence o bytes that comprise each individual packet, as well as a dissector

view which shows what the packets contents actually mean rom the point o view o variousInternet protocol layers. A careul examination o this evidence could involve a byte-or-byte,packet-or-packet comparison to determine exactly which packets were spooed. I one isperorming such a comparison, it is important to appreciate that Wireshark captures more thanjust the IP packets that get transmitted over the Internet; each IP packet is typically capturedwrapped inside a link-layer header (oten reerred to as an Ethernet rame header or Ethernetpacket header on an Ethernet-style network, including a wired Ethernet or wi network). Link-layer headers are used to communicate between computers on the same local Ethernet networkand are discarded and regenerated whenever a packet is orwarded through any router. Tus,there is no reason to expect link-layer headers to correspond between two packet traces un-less those packet traces were captured on the same physical local-area network. When manu-

ally perorming a detailed packet comparison, packets should be considered the same i theInternet Protocol headers and payload match up; the link-layer headers can simply be ignoredbecause they were never orwarded over the Internet.

Even some discrepancies between IP headers are to be expected; or example, the ime-o-Live(L) eld is supposed to be decremented by each router that orwards a packet, so the Lvalue when a packet is received should always be smaller than the original L value when itwas transmitted. Similarly, the IP checksum eld that indicates whether a packets IP head-ers were transmitted without error has to be recalculated each time the packet is orwarded,because the L value has changed. Tere are also other circumstances in which an ISPsrouters may, consistent with Internet protocol standards, legitimately alter other elds in the IP

headers. 12

On the modern Internet, there are usually several ISPs involved in orwarding packets romtheir source to their destination. As a technical matter, any o these ISPs has the ability tospoo (or drop) packets. Detecting the presence o spooed packets, then, does not directlyreveal or conclusively establish which ISP was responsible or injecting them. o nd outwhich ISPs were involved in the process o orwarding packets rom one computer to another,one can use the traceroute tool or any o its descendants or variants. (On Unix and MacOS,the command-line version o this tool is generally called traceroute; on Windows, it is knownas tracert.) Tis tool is run with a host name or IP address as its argument and experimen-tally determines the path probably ollowed by probe packets, displaying a list o the routers in

between the local computer and target computer. In order to establish which ISP is respon-

11 o compound the suspiciousness o this situation, each machine believed that it was not the rst to send aRS packet in this transaction the local and remote machines each believed that the other party had initi-ated the process o disconnecting the communication. It can be legitimate in the CP protocol to send aRS in response to a RS, and this trace suggests that each machine believed that that is what it was doing disconnecting only ater the other end had already disconnected.

12 We might compare this, albeit imprecisely, to the behavior o the post oce in delivering a letter: the post ocemay apply a postmark or print a bar code on the outside o the envelope, stamp delivery-related notations on it, oreven correct an erroneous postal code. However, the post oce is not supposed to alter the contents o letters.



sible or packet spoong, one could try experiments with a wide variety o ISPs in order todetermine i users o one ISP routinely or disproportionately experience that sort o spoong;one could also try to enable a virtual private network (VPN) to hide trac rom a local ISP,although the details o this process are beyond the scope o this document.

ui pp fl

As weve described above, packet capture les produced by Wireshark and many other network

analyzer programs are normally in thepcap ormat (also known as libpcap ormat or tcpdumpormat). Tis ormat is an open standard that is widely understood by network analysis sot-ware. I youre certain that a pcap le or set o pcap les youve made does not contain sensitivepersonal inormation recalling our earlier warning that it may, by deault, contain a record oall Internet activity your computer perormed while the capture was active you can considersharing pcap les with an ISP as part o a problem report or support request, or publishingthem on a blog or web site when documenting a problem youve experienced. Tese les areconcrete, useul evidence that can help technically knowledgeable people diagnose networkproblems and conrm that a problem such as packet spoong by an ISP is really occurring.

EFF is developing a tool calledpcapdifto help automate the process o comparing large pcap

trace les that were made simultaneously at each end o a connection. Tis automation makesit easier to nd packet spoong or tampering when one doesnt know what to look or ahead otime and could make the comparison process less tedious (considering that many communica-tions involve thousands o packets or more, and not all tampering will have results as obviousas CP RS injection). Te pcapdi program is a command-line utility written in Pythonwhich requires exactly two pcap packet traces; it automatically compares them to nd discrep-ancies indicating dropped and spooed packets. You can download pcapdi rom http://www.e.org/testyourisp/pcapdi. Your computer must already have the Python interpreter in-stalled; i not, you can obtain it rom http://www.python.org/.

Currently, pcapdi also requires the pcapy module rom http://oss.coresecurity.com/projects/

pcapy.html in order to read pcap les; binary versions o this module are available or Windowsand Linux, but it must be compiled rom source code or MacOS. Future versions o pcapdimay be able to run without pcapy.

pcapdi is run rom the command line; you must speciy two pcap trace les and the local IPaddress o the computer where each was captured. In addition to producing statistics aboutoverall rates o packet dropping and spoong, pcapdi will produce a list o the IP identica-tion eld values o each such packet. Tis can help you locate packets o interest in a programlike Wireshark much more quickly. For example, i pcapdi says that a packet with IP iden-tication value 4321 was spooed in the outbound direction, you can enter the display lterip.id eq 4321 into Wiresharks display lter eld and see only the packet (or packets)

with this particular IP identication value. ypically, IP identication values uniquely identiyIP packets transmitted by a particular computer, although these values will be repeated at leastevery 65536 (216) packets.

Observing a very large number o orged packets may suggest that something is systematicallywrong or example, you may have started or ended the captures at dierent times (so that onemachine has no record o having sent many o the packets that it did, in act, send), you mayhave limited the number o bytes captured per packet at one end but not at the other end, youmay have some kind o protocol ofoading active on one end (so that one computers operat-ing system is wrong about the contents o the packets that the network card actually sent over
http://www.eff.org/testyourisp/pcapdiffhttp://www.eff.org/testyourisp/pcapdiffhttp://www.python.org/http://oss.coresecurity.com/projects/pcapy.htmlhttp://oss.coresecurity.com/projects/pcapy.htmlhttp://oss.coresecurity.com/projects/pcapy.htmlhttp://oss.coresecurity.com/projects/pcapy.htmlhttp://www.python.org/http://www.eff.org/testyourisp/pcapdiffhttp://www.eff.org/testyourisp/pcapdiff


17/17

the wire), you might still have NA or a rewall enabled at one end o the connection, or yourISP might be orging or altering packets routinely, perhaps with a technique like transparentHP proxying that is invisible to most application sotware. It is important to think criti-cally about the signicance o evidence and whether results are reproducible or attributable toconounding actors. pcapdi and the techniques described here are meant to help users ndanomalous discrepancies between packets sent and packets received, not to denitively assignblame or the source o those anomalies.

Li m tcP/IP d ipi w p

For a deeper understanding o network protocols and packet traces like those described here,you can consult the Internet standards documents that speciy the CP/IP protocol suite.Most o these documents have been published in the RFC document series available at http://www.rc-editor.org/ and http://www.aqs.org/rcs/. For example, the Internet Protocol (IP)is described in RFC 791, the ransmission Control Protocol (CP) in RFC 793, and the UserDatagram Protocol (UDP) in RFC 768.

An excellent guide to the CP/IP protocol suite in book orm is W. Richard Stevens, CP/IPIllustrated: Volume 1, Te Protocols (Reading, MA: Addison-Wesley, 1994), ISBN 0201633469.

Stevens uses the tcpdump tool to produce packet traces on 1990s-era Internet-connectednetworks, and careully explains the theory and practice o Internet networking with reerenceto these packet traces. Doing the same thing with Wireshark might be clearer today becauseWireshark has a riendlier interace and more extensive protocol dissection than tcpdump, butStevenss explanations are clear, thorough, and generally valid or the Internet o today.

awldmTanks to Chris Palmer and Karl Fogel or their comments.
http://www.rfc-editor.org/http://www.rfc-editor.org/http://www.faqs.org/rfcs/http://www.faqs.org/rfcs/http://www.rfc-editor.org/http://www.rfc-editor.org/

Documents

Packet Injection