Michigan HTCIA March 29 th, 2006 Usenet Abuse Primer

Michigan HTCIA

March 29th, 2006

Usenet Abuse Primer

Introductions

• Mark Lachniet ([email protected])• Technical Director, Security Services Group• Responsible for technical oversight of all security

offerings including live and static forensic analysis• Certified Information Systems Auditor (CISA)• Certified Information Systems Security Professional

(CISSP)• Frequent presenter at local educational conferences

(MACUL, MAEDS, MIEM)• Technical certifications from Novell, Microsoft, Linux

Professional Institute, etc.• Formerly the I.S. Director at Holt Public Schools• Also an instructor for Walsh College’s MSIA program• NOT a Law Enforcement Officer - YMMV

Introductions

D/Sgt. Michael Harrington,CFCE, EnCE

• Forensic Analyst Michigan State Police

• Past President MIHTCIA

• HTCIA International Treasurer 2004

• HTCIA International Secretary 2005

• Gentoo and Sam Adams Enthusiast

• Email:[email protected]

Warning!

• Due to the subject matter, there may be offensive content in this presentation

• No graphic images will intentionally be displayed, but be warned, there is definitely disturbing content out there

• You have the capability to do the hands-on exercises, but as time is very limited this will be on a “do it if you can keep up” basis

• The dev server (dev.lachniet.com) will be available for testing after this presentation. If you would like to use it, please send me an e-mail. I cannot guarantee how long I will keep the server up, but its free

Agenda

• Usenet overview• NZB files• Finding Usenet content• Usenet anonymity• Legal liability• Yenc encoding• Investigation• Demonstration(s)

Why do we care?

• Usenet is very commonly being used for piracy and pornography, and very little information about it is generally known in the Law Enforcement community

• There may be legal liability for employers• Usenet is widely distributed and heavily used• Usenet can be extremely difficult to

investigate – anonymous services that market based on not logging are commonplace

• Usenet content, particularly the Yenc format of binary encoding, is not well supported by conventional forensic tools

The Paper and Future Training

• This presentation is based on a whitepaper, developed by Mark and Mike, which will soon be released

• We are considering putting together a longer training session based on this content (2hrs isn’t enough)

• We would be interested in both feedback on the paper prior to release, and in a venue for doing a longer training session

• Contact me after the session or at [email protected] if you are interested

The History of Usenet

• Usenet is a distributed Internet discussion system that evolved from a general purpose UUCP network of the same name. Users, sometimes called Usenetters, read and post email-like messages (called "articles") to a number of distributed newsgroups, categories that resemble bulletin board systems in most respects. The medium is sustained among a large number of servers, which store and forward messages to one another. Usenet is of significant cultural importance in the networked world, having given rise to, or popularized, many widely recognized concepts and terms such as "FAQ" and "spam".

Usenet Today

• Usenet is like a bulletin board – users can post messages which can then be read by other users

• The messages themselves generally resemble a standard e-mail

• Newsgroups such as alt.binaries.games are used to organize the messages by topic, and are the “buckets” in which the messages are stored

• Usenet servers talk amongst themselves to share messages and synchronize

• Thus, if you post a message to alt.binaries.games on a commercial server such as Giganews, it will soon be replicated to other servers around the world

• Refer to RFC 0977 and RFC 1036 for more technical details

Usenet Groups

• There are a variety of newsgroups for just about anything you could ever want or need. As of 3/26/06 there are 105,228 groups carried by Giganews.com

• Groups that are distributed worldwide are split into seven classifications: comp, misc, news, rec, soc, sci, and talk

• Conversely, the alt tree of Usenet is anarchy incarnate, and has less oversight

• For example:– alt.adjective.noun.verb.verb.verb – alt.american.olympians.choke.choke.choke – alt.christnet.bible-thumpers.convert.convert.convert– alt.binaries.games– alt.binaries.pictures.erotica.early-teens*

Example Usenet Message Headers

Path: border1.nntp.dca.giganews.com!nntp.giganews.com!feed2.newsreader.com!newsreader.com!npeer.de.kpn-eurorings.net!news.tele.dk!news.tele.dk!small.news.tele.dk!news.astraweb.com!newsrouter-eu.astraweb.com!eweka!hq-usenetpeers.eweka.nl!81.171.88.219.MISMATCH!newsreader30.eweka.nl!not-for-mail

From: "Apollo" <[email protected]>Subject: were can i download the series?Newsgroups: alt.binaries.battlestar-galacticaDate: Sun, 1 Jan 2006 16:47:29 +0100Lines: 7Message-ID: <[email protected]>Organization: Eweka Internet ServicesNNTP-Posting-Host: Eweka Internet Services X-Trace: Posted by Eweka Internet Services, http://www.eweka.nl X-Complaints-To: [email protected] Xref: number1.nntp.dca.giganews.com alt.binaries.battlestar-

galactica:824361

Example Usenet Message Body

This message is in yEnc format. If your newsreader cannot display this message,

please visit http://www.ydecode.com/ and download yEnc decoder.

=ybegin line=128 size=24064 name=hello.docúù;=JËÛD

****************h*-*()3*0***********+***T********:**V***+***()))****S***)))))))))))))))))))))))))))))))))))))))))))))))

))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))

))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))

))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))

)Ïë*©Š3=n**<é******:*****0**m2**8*Œ”Œ”±±******************3=n@*X:**®**®**m*******************************))9*********))9**

Similarities to e-mail Messages

• There are several similarities to regular e-mail messages

• We have a From: heading and a Subject: as well as a Date: and other similar looking headers

• However, some key differences exist. For example, rather than a To: field, we have a Newsgroups: field

• You can’t really trust the From: field to be an actual person – it is easy to spoof. Only a complete idiot would post illegal content with their real e-mail address.

• One thing you can *probably* trust is the Message-ID field. This field shows you what news server originally received the message. Although this could also be spoofed by setting up your own Usenet server, the common criminal probably isn’t that smart.

Usenet Message Propagation

• In layman’s terms, my interpretation, Usenet propagation works something like the following:

• A client with access to a Usenet server posts a message. This communication is between the client and the server directly, and usually takes place over TCP port 119. Let us suppose that the client had posted their message to alt.binaries.battlestar-galactica.

• The server receives the message and assigns it a unique Message-ID field.

• The server communicates with its Usenet peers (other Usenet servers). For each message that it has received locally, it checks with its peers to see if any of them subscribe to the alt.binaries.battlestar-galactica groups.

Usenet Message Propagation

• If the peer servers do subscribe, they communicate between themselves to see if they already have a copy of the message. This is based on the Message-ID field. If they subscribe to the newsgroup but do not have the specific message, it is transmitted to the peer. If the server already has the message, it ignores that message.

• A second client (the consumer) connects to their Usenet server (most likely a server in a completely different part of the world). They pull up a list of messages for the newsgroups for which they are subscribed, and the message in question is listed.

• If the consumer client wishes to download the message, they do so, and the content is transferred to their computer.

Usenet Binaries

• Just as in e-mail, it is possible to encode a binary in a text format and transmit it

• Some methods to encode binary content include BASE64, BinHex, UUencode, Quoted Printable and the Usenet-specific yEnc format.

• More information on this will be included in Mike Harrington’s Forensic Analysis (later on)

• In the case of large files such as CD Images, video content, etc. the limit 10,000 lines of text is often exceeded

• In some cases, a binary file may fit within the limit of a single Usenet message. For example, small images and smaller files may be entirely self-contained.

Demonstration – A Small Binary

• A small repository of files to do the exercises can be found at http://dev.lachniet.com

• First we’ll install NewsReactor and subscribe to our test groups

• Download the install from http://dev.lachniet.com, or http://www.daansystems.com/newsreactor/, and run it

• Click next a bunch of times• We’ll then configure our dev Usenet server

and subscribe to the groups

Configuring the Usenet Server

• Go to File-> Options and add the server

The server is dev.lachniet.com

The server doesn’t require authentication – it’s wide open for now

Subscribe to Groups

• Ignore my accidental extra groups

• Click on the Groups tab, and then click on the “newsserver” button to download a list of groups on the server. Select alt.binaries.mihtcia.naughty

• Click OK

View the Contents of the Group

• Click on the alt.binaries.mihtcia naughty link

View Raw “inappropriate” Message

• Double-click on the last message to pull up the raw text

• This is what you might find if you analyzed a NNTP server, or a client that had downloaded the mssage

View the Decoded Binary

• Close the window and this time right-click on the message and select “grab and open”

• Hey, it was the 70’s

Multi-Part Postings

• There is a maximum number of lines, so large files will be broken up when posted

• Some clients (e.g. Outlook) won’t reassemble them, while smarter clients (e.g. NewsReactor) will

• This is *different* from multi-part archives, which we’ll talk about later

• This is due to the maximum number of lines that are allowed in a Usenet message

Multi-part Postings Example - OE

• You can see a single file broken into 9 chunks in OE – note 517kb per message

Multi-part Postings Example - NR

• NewsReactor aggregates and combines them for you

Multi-part Archives

• Splitting large binary postings (such as a CD ISO image) into smaller files is useful, but not convenient and fault tolerant

• For example, if you lose any single part of the posting, you will not have a valid binary and will have to download it again

• This can happen if a part is lost, corrupt, or never uploaded

• It is better to break a large files into multi-part archive files of a more manageable size such as 4mb.

• That way, if part of it is corrupt, you only have to download 4mb instead of 650mb

• The most popular way to split up large files is to use WinRAR

Multi-part Archive Example

• Look at the Babylon 5 Videos posting:

Multi-part Archives - WinRAR

• In the above example, if you successfully download all of the RAR files (1-9) you can then open the archive and uncompress it

• This is how most software and pornography is distributed currently

• Also note that NewsReactor gives you a high-level title heading for the entire series of files, and allows you to download them all with a single click

• Lets demonstrate this…

Demonstration – Multi-part Archives

• Select the high-level message, right click and hit “grab” and you can see it downloading the parts


• Now click on “browse” to open the directory they were downloaded to

• Make sure WinRar is installed (get it from dev.lachniet.com if it isn’t)

• Double-click on the first file (b5cd.part01.rar) and you’ll see the contents of the combined RAR file


• Click “Extract to” and select c:\

• Navigate to c:\b5cd to view the files

• Double-click on “loader.exe” if you have an unhealthy interest in science fiction and mythology

Parity Archives

• Multi-part archive files are all well and good, but the Usenet (let alone the Internet!) is not a reliable medium for transmitting large amounts of data.

• In particular, with the Usenet, it is common for some parts of a large archive file to be lost (generating a “fill” request for the poster to resend the missing parts).

• To accommodate this problem, enterprising software engineers came up with a way to create parity files

• This parity system is very similar to a RAID-5 disk array in the hardware world.

• The long and short of it is that with PAR and PAR2 software, it is possible to recreate complete archives, even if some portions of an archive are missing.

• I used WinPAR (on dev.lachniet.com) to create parity archives for the Babylon 5 videos

Demonstration – PAR files

• Navigate to your hard drive where all of the RAR files are stored.

• Delete 1 of the 9 archive files and attempt to open the archive

• You’ll still be able to see what the contents of the file are, but when you try to extract it to c:\ you’ll get an error


• Now, double-click on the PAR2 file “b5cd.vol01+02.PAR2”


• You can now “reconstitute” the missing file from the PAR files by clicking on “repair”

• At this point you could then open the RAR files and uncompress the archive

• There are several implications for LEO’s:– Evidence may exist on the hard drive but not be

findable until you combine multi-part postings and/or multi-part archives

– Searches for yenc strings and archive files (not just RAR but also ZIP, PAR, PAR2, etc.) should be included in all searches

– Even partial files (one rar out of many) will still have a directory table, so you may be able to get interesting search terms out of it

NZB Files

• Since each Usenet message is unique, and has a unique identifier such as Message-ID: <[email protected]> it is possible to create a type of index file that makes it easy to download binary content

• NZB files are specifically designed for this purpose

• Many programs and search sites will automatically create NZB files for you, so you don’t have to find the content the hard way

NZB File Example

<?xml version="1.0" encoding="iso-8859-1" ?><!DOCTYPE nzb PUBLIC "-//newzBin//DTD NZB 1.0//EN"

"http://www.newzbin.com/DTD/nzb/nzb-1.0.dtd"><nzb xmlns="http://www.newzbin.com/DTD/2003/nzb"><file poster="mike <[email protected]>" date="1135277730"

subject="MISC OLDGAMES gone FREEWARE vol 3 - File 06 of 23 - yEnc "cyclones.zip" 6736639 bytes (01/17)">

<groups><group>alt.binaries.old.games</group></groups><segments><segment bytes="416782"

number="1">[email protected]</segment>

<segment bytes="417226" number="2">[email protected]</segment>

</segments></file></nzb>

NZB File Example

• In the previous example, we can see that the binary is “MISC OLDGAMES gone FREEWARE vol 3” and is 6,736,639 bytes

• We can tell that the original server to receive the posting was 194.152.65.251

• We can see a series of unique message ID’s that the server assigned (e.g. Xns9734CAA2DE2A8orisitdunnocom)

• Using a modern newsreader, we could simply open that NZB file and start downloading

Finding Usenet Content

• Since there are so many newsgroups, and so many messages, it is sometimes difficult to find specific content

• Just the headers on an active newsgroup can be gigabytes in size

• Fortunately, there are a number of online search engines that can be used to narrow down what you are looking for

http://groups.google.com

• Intended for text searches• Allows you to search a LOT of historic data,

going back (at least) into the early 90’s• This is also a great site for doing background

checks on people who have been around for a while

• In 1992, nobody ever heard about archiving, and probably were a little more loose with their postings than they would be nowadays

http://www.yabse.com/index.php

• Will find binary content and make a NZB for youb

• You can use a URL such as http://www.yabse.com/index.php?q=freeware

http://www.guba.com

• Guba provides you not only with a search engine for binary content, but a means of directly downloading it

• You don’t need to use a news client at all – just get a subscription to Guba, and search for the content you want

• You can then simply download the complete binary from the web site

• You can also convert it to a different format, so you can watch it on your iPod or PSP

http://www.guba.com

Usenet Anonymity

• As previously mentioned, it can be tough to track down a Usenet poster even under the best of circumstances

• Names, servers and message IDs can all be spoofed• In addition, a number of Usenet service providers make a

business out of running systems that intentionally don’t keep any logs

• This could make it very tough to catch the person who already posted, but you might be able to catch them if they do it more than once

• Example Privacy policy (from easynews.com)– Easynews takes your privacy seriously. We have one of

the most aggressive privacy policies in the industry. – Easynews does not monitor or log downloads. – No identifying information is placed in your Usenet posts.

Your posts are virtually anonymous with all X-Headers removed.

Usenet E-Mail Gateways

• There are also allegedly services that will forward between Usenet and E-Mail

• The software for this is available, but I don’t know of a commonly used service

• This might further obscure forensic evidence, especially when combined with anonymous remailers

Legal Liability

• As an employer or ISP, you may have some liability in regards to Usenet

• What about employees downloading warez or porn from work?

• What about hosting a NNTP server without proper access control? What if someone uploaded illegal material to it?

• Certainly the usual “sexually charged workplace” issues apply here

• The Business Software Alliance could also take you to task for pirated software

Investigation – on the Client

• There may be traces of Usenet files on a workstation that you could turn up, either in allocated space, or in deleted / slack space.

• It would behoove a forensic investigator to include looking for evidence of Usenet abuse in their standard operating procedures.

• Keyword searches might include:– usenet– nntp– news– binaries (or alt.binaries)– known NNTP server names and IP addresses (you might

find a personal firewall log or something with entries to a Usenet server, even if the program was deleted)

– Yenc strings (see later slides)

Investigation – on the Client

• File searches might include:– .RAR archive files

– .PAR and .PAR2 parity files

– .NFO description files

– .NZB batch files

– .ZIP files

– Known NNTP clients (in file space and the registry)

– Known Usenet search engines in browser caches (Guba, etc.)

Investigation – On the Network

• If you had access to the network of either a suspect ISP or a suspect computer, you could use a protocol analyzer to identify suspicious activity.

• As previously noted, it is possible to get Usenet binaries entirely over web connections using gateways, as well as by watching for actual NNTP traffic.

• Using firewall logs, or a protocol analyzer such as Ethereal you might look for connections such as:– HTTP / HTTPS (TCP port 80 and 443) connections to known

Usenet web servers such as Guba– NNTP connections to any host on TCP port 119– NNTP over SSL connections to any host on TCP or UDP port 563– Traffic with a payload matching the keywords listed previously

• Of course, a crafty criminal will use the NNTP over SSL encryption option, or tunnel all of their connections through a SSH tunnel or something.

Investigation – At the Provider• Theoretically, you can get a provider to help you with a court

order of some kind. However, as we noted previously, a lot of them don’t keep any records at all, so this may be difficult

• If you are going to get any information at all from a provider, you’ll need to have that unique Message-ID field to work with.

• It is relatively certain that if you see a message with a Message-ID such as “[email protected]” that the machine with the IP address of 194.152.65.251 was the one that originally took the posting

• You might also look at the PATH header. For example, our previous example had a path of:

• Border1.nntp.dca.giganews.com!nntp.giganews.com!feed2.newsreader.com!newsreader.com!npeer.de.kpn-eurorings.net!news.tele.dk!news.tele.dk!small.news.tele.dk!news.astraweb.com!newsrouter-eu.astraweb.com!eweka!hq-usenetpeers.eweka.nl!81.171.88.219.MISMATCH!newsreader30.eweka.nl!not-for-mail

• This might help you find a person with logs in the event of a spoofed Message-ID

Legal Citations

• Coming soon! We are hoping to do a review of cases for citable precedent

yEncode in Computer Forensics

• What is yEncode?– Encoding scheme for transmitting binary

information in email and newsgroups– yEncode takes advantage of the entire 8-

bit character set resulting in output only 1-2% bigger than the original binary(compare to 40% for traditional 7 bit encoding)


• Header– Single yEncoded binaries always begin with a

header that contains an escape character(‘=‘), the keyword ‘ybegin’, and followed with parameters ‘line’, ‘size’ and ‘name’ as in the following example

=yenc line=128 size-123456 name=mybinary.dat

The filename must always be the last item on the line.

• Trailer(footer)– Always begins with escape character ‘=‘

and ‘yend’ keywordand MUST contain the size of the original unencoded binary(in bytes) as in the below example

=yend size=123456


• Verifying Integrity– yEncoded documents may include a 32 bit

CRC value in the trailer to held decoders evaluate the integrity of the binary as seen below

=yend size=123456 crc32=abcdef123



• Multi-part files– Due to size binaries are frequently split

into multiple parts for transmission– This results in frequent unusable binaries

due to missing parts and/or data corruption.

– Multi-parts have standard ‘ybegin’ line and then additional keyword ‘part’ in header


• Multi-part files (cont’d)– The keyword part specifies part number

and identify it as a multi-part file– If ‘part’ is included an additional ‘ypart’

keyword line must follow which specifies the information about the part


• Multi-part files(cont’d)– ‘ypart’ keyword requires a being and end

keyword specifying the information about the part

– The file must end with a modified ‘ypart’ trailer line-an additional ‘part’ is added to specify the part number and must match the original one in the header.


• Multi-part file (cont’d)– The trailer in a multi-part file must also

contain a ‘pcrc’; keyword representing the CRC32 value of the preceding encoded part. It is also possible to encounter a CRC value for the entire encoded binary.

– The ‘size’ keyword in multi-part trailers represents the size of the file part not the entire file


• Multi-part files (cont’d)– To verify integrity a decoder must re-

compute ‘begin’ and ‘end’ values in the ‘ypart’ line.

– If the expected part size differs from the part size in the ‘yend’ line the file is corrupt.

=ybegin part=1 total=10 line=128 size=500000 name=binary.exe

=ypart begin=1 end=100000



Suggestions for Post-Mortem Analysis


• Our scope is going to be limited to three more popular newsgroup applications– Outlook Express– Mozilla Thunderbird– Free Agent from Forte


• Outlook Express– EnCase

• Encase will search for and automatically decodes OE DBX files.

• EnCase will separate attachments for viewing-but will not reassemble multi-part files

• Pictures in newsgroups MUST be viewed in the email view they do not show up in the gallery

• Bookmarking must be done from the email view


• FTK– FTK will not extract newsgroup DBX

files into an easily readable format– FTK will show the yEncoded binary so

you can copy out the binary for further processing


• Thunderbird– FTK

• There is currently no implementation in FTK to read newsgroups from Thunderbird

– EnCase• The same applies in Encase for Thunderbird

as applies for OE.

• Again, you must view attachments in the email view and bookmark there as well


• Forte Agent– FTK

• There is currently no implementation in FTK to read newsgroups from Forte Agent-however like dbx files the yENC encoding can be seen and exported for analysis

– EnCase• The same applies in Encase for Thunderbird as applies

for OE.

• Again, you must view attachments in the email view and bookmark there as well


Manual Decoding of yEncoded

Binaries


• Yenc32 (www.yenc32.com/download.php)– Free (yes, I said “free”) decoder that can be

used with Outlook Express, Thunderbird and Forte agent as well as other newsgroup readers

– Standalone program as well as right click integration with the gui

– Tutorials on how to decode the above available online (http://www.yenc32.com/support.php)


• Email Examiner (Mailbag Assistant)– Will read the downloaded newsgroup messages

but will not decode the attachments (All common mailbox types-OE Thunderbird and Forte agent).

– EMEX will tell you the status of the message (i.e. whether or not the binary still resides on the server or has been downloaded)

– To decode with EMEX export the emails as a generic mailbox with the extension “.yenc” Right click on the file and decode with yenc32.


• Linux– Convert DBX files to mbox format and import

into a mail reader of choice. Decode using a yEnc decoder for Linux.

• http://www.yenc.org/linux.htm

– For tips on how to decode email to a flat mbox format please see the whitepaper “Analysing Exchange and mbox emails using open source software” at

• http://www.forensicfocus.com/computer-forensics-papers


• Thunderbird– Thunderbird as a newsgroup reader

supports yEnc binaries(though it will not reconstruct multi-part files)

– Convert dbx mailboxes to an mbox format ( as outlined in the aforementioned paper) and import into Thunderbird. Export your binaries.


• Searches– Regular expression or keyword searches

• ‘=yenc’ or ‘=yend’ or ‘=ypart’• ‘name=xxx’ if the name of the suspect binary

is known• Use a tool such as Foremost (Linux or

Cygwin), SMART or Data lifter with the ability to carve using header and footer-note this will not get CRC values or size since these values can be optional in the yEncoding standard


Questions and Comments

• Thanks!

Documents

Michigan HTCIA March 29 th, 2006 Usenet Abuse Primer