Infrastructure Tracking with Passive Monitoring and Active Probing: ShmooCon 2015 Presentation

Infrastructure Tracking with

Passive Monitoring and

Active Probing

Anthony Kasza Dhia Mahjoub

January 18th, 2015

November 11, 2014

Hello,

I am a security researcher at OpenDNS. I have been tracking the movements of the Gameover Zeus (GOZ) botnet. Your registrar has been used to register domains used for command and control communications between the operators of this botnet and compromised hosts. Are you able to collaborate in tracking and shutting down these domains?

-AK

Registrar Abuse Desk Response Times

Webfusion 1hr 44mins

Enom 2hrs 36mins

Namesilo 21hours 27mins

Bigrock Solutions 2days 1hr 20mins

TodayNic 1 week

101 Domain -

Active Registrar -

Melbourne IT DBA internet names worldwide -

The Registry at Info Avenue -

Turncommerce DBA Namebright -

Speakers

@dhialite

Senior Security Researcher

DNS, networks, data analysis, threat detection, graphs

@anthonykasza

Security Researcher

DNS, network protocols, threat detection, Bro IDS

github.com/anthonykasza

Agenda

Importance of Threat Intelligence

Active Probing

Passive Monitoring

Fastflux Case Study: Zbot

Tracking System Overview

DGA Case Study: newGOZ

Tracking System Overview

Conclusion

OpenDNS’ world network

STUB

CLIENTS

RECURSIVE

NAME SERVERS

AUTHORITATIVE

NAME SERVERS

root

tld

domain.tld

~2 TB of query logs

per day, compressed

Types of DNS traffic

Threat Intelligence

Relevant, timely, and useful information that helps

take action (strategic, or tactical)

Examples of tactical actions (not an exhaustive list)

-Blocking known malicious domains, IPs

-Preemptively block suspicious domains, IPs

-Further investigate domain patterns, IP infrastructure

-Further investigate malware samples, anomalous

traffic patterns

Network Intelligence Collection

Techniques

Active Probing

Active Probing

Current state, RIGHT NOW

thing being investigated

thing’s neighbors

Direct - touch the thing being investigated

Indirect - ask around about the thing

Active Probing: Direct

-Port scan, service banner grabs (shodan/nmap/masscan)

e.g. hosting Angler EK, sharing identical server setup

-Collect content (http/ftp)

noisy – is detectable

block by source or return misleading content

64.251.7.239 – 64.251.7.241

22/tcp open ssh OpenSSH 5.3 (protocol 2.0)

80/tcp open http nginx 1.6.2

Active Probing: Indirect

DNS

Domain to IP, Domain to Name server, Name server to IP

BGP and IP whois

IP’s ASN and upstream ASNs

Explore sibling ASNs

hosting provider

Domain whois

domain, authoritative name server domain

registrar, registrant, created/updated/expire times

Active Probing: IndirectQuery for DNS records

-Domain to IP,

-Domain to Name server,

-Name server to IP,

Can be considered direct (i.e. noisy & trigger alerts) if

authoritative name servers are operated by same bad actors

Scalable tools:

adns http://www.gnu.org/software/adns/

Massresolver https://github.com/jedisct1/massresolver

http://www.gnu.org/software/adns/

https://github.com/jedisct1/massresolver


Query for BGP and IP whois data

-IP to ASN, Team Cymru, or routeviews + PyASN

-Upstream and sibling ASNs (SPN concept, BlackHat 2014)

-Hosting provider: rogue, lax or abused

e.g.

http://www.serverpronto.com/ US

https://king-servers.com Russia

http://www.mach9servers.com/ US

https://www.bacloud.com Lithuania

http://www.qhoster.bg/ Bulgaria, reseller, register domains & hosting

http://www.serverpronto.com/

https://king-servers.com

http://www.mach9servers.com/

https://www.bacloud.com

http://www.qhoster.bg/



Both ranges belong to Serverpronto, hosting subdomains

injected under compromised GoDaddy domains to serve EK

64.251.7.239 – 64.251.7.241

22/tcp open ssh OpenSSH 5.3 (protocol 2.0)


64.251.22.201 – 64.251.22.207

22/tcp open ssh OpenSSH 6.0p1 Debian 4+deb7u1 (protocol 2.0)


111/tcp open rpcbind 2-4 (RPC #100000)




All SPN ASNs except one ASN has a downstream adjacent ASN

-AS47145: compromised IPs hosting zbot FF CnC domains





Domain whois

-Domain, authoritative name server domain

registrar, registrant, created/updated/expire times

problems

daily changes are often too coarse

client provided information isn’t always accurate

Tools: whois client, scrape web-based whois sites,

commercial offerings

Domain Registration Terms

Registrants

Reseller

Registrar

Registry

NS RR

NS RR

Contact Info

Passive Monitoring

Passive MonitoringPrevious state of things or patterns derived from behavior monitoring

Passive DNS reconstruction

pivot from a seed

domain -> IP -> domain

domain -> nameserver -> domain

Correlation via registrant email -> reliable in specific cases

Client query patterns

domain lexical analysis

query spikes

query co-occurrences

Correlation via malware samples, domain, IP artifacts

Application layer data (sinkhole)

Combination of interchangeable models

FF model, sample network report, DGA model, traffic pattern

model, any others

Pivot around artifacts (domain, IP, sample

features, traffic features, co-

occurrences, etc.)

Apply filtering heuristics to remove FPs (traffic pattern,

lexical features, etc.)

New domains, IPs can do a feedback loop

D

D

D

IP

I

P

D

D

D

D

IPs

Domains

Domains


Pivot from a seed

D

D

D

NS

D

D

D

D

NSs

Domains

Domains


NS

Pivot from a seed

Domain detected by traffic or malware

analysisGet registrant email

Extract all domains registered by same

email

Apply filtering heuristics to remove

FPs (traffic, subdomains, resolution,

url patterns, etc)

Correlation via registrant email

Correlation via registrant email

-Effective for compromised domains registered

by same registrant email

Injected with subdomains for EK, browlock, etc.

e.g. GoDaddy compromised domains

-Effective for malware dedicated CnC domains

e.g. GOZ, zbot, Tinba

[email protected]


Client IPs

DomainsIP

IP

IP

D

D

D

IP

Time window


Co-occurring domains

• Temporal proximity of domain lookups

• Bipartite graph of client IPs to domains during a short

time window

• Consider both resolving queries and nxdomains

• Use cases of interest:

botnet CnC domains especially DGAs

Domains sharing same theme, campaign, e.g. carding sites,

click-fraud, etc.

Compromised sites leading to EK or malware domains


Pivot from seed sites, e.g.

e.g. seed list of carding sites (monitoring during Target breach)

carderprofit.cc, carder.su, cardersunion.net, cardingworld.cc,

cclub.bz, cclub.su, clubr.ru, crdclub.ws, darkmarket.ws,

dumps4you.cc, infraud.su, jworldtopcc.su, lampeduza.so,

proclub.ws, prov.cc, unclesam.vc, validcc.su, verified.ms, vpro.su

Heuristics:

Domain -> hosting IP -> Domain

Domain -> client IP -> Domain (co-occuring domains)

Domain -> name server -> Domain

+ filtering heuristics to remove FPs


Client query patternsSome extra carding and stolen credentials sites discovered (there are a lot more):

prvtzone.cc

best4best.su

cardrockcafe.so

cardrockcafe.cc

cvv.me

d4rksys.cc

ssndob.cc

ssndob.so

torcvv.cc

darkmoney.cc

vini.cc

uniccshop.ru




Domain detected by traffic monitoring (FF, DGA, other models)

Get malware sample analysis report

Extract queried domains from network

traffic report

Apply filtering heuristics to remove

FPs (traffic, subdomains, resolution,

etc)

Correlation via malware network artifacts

Correlation via malware network artifacts

Some filtering heuristics:

-Similar traffic patterns (e.g. spikes or shape of traffic curve)

-Similar domain lexical features

-Similar subdomain and hosting IPs patterns

-Similar website content

-Similar url patterns (3rd party analysis report, sinkhole, own sandbox)

…

Open sources for analysis reports:

VirusTotal, totalhash, malwr, ThreatExpert, Sophos and Microsoft

threat reports

Web-scraping malware samples & reports

Sources:

-VT, totalhash, malwr, ThreatExpert, Sophos and

Microsoft threat reports

-Use commercial version

-Scrape online reports using free open proxies to

prevent throttling or blocking of your source IP

Application layer data (sinkhole)

-This could arguably be active…

-Application layer data validation

-Get url patterns for sinkholed domains

-Or get urls from VirusTotal, totalhash reports, etc.

-Use ET signatures to match against traffic

Other sources of Intel

-Good old google, other search engines

-Reliable friends, colleagues

-The infosec community

Automation

Scale

Accuracy

are crucial

+ Human Validation

Fast flux case study:

Zbot proxy network

• DNS-based redundancy/evasion technique

• Fast flux domain resolves to many IPs, many ASNs,

many CCs, relatively low TTL

• Fast flux domain resolves to 1 IP with TTL=0

• Ex : Trojan CnCs, spam, scam, pharmacy, dating domains

Fastflux definition

(1) Initial list of zbot fast flux

domains

(2) Get IP, TTL via direct lookup into

DNSDB

(3) Extract IPs s.tTTL=150

(4) Get domains from IPs via

inverse lookup

(5) Add domains from (4) to list (1)

(6) Extract IPs s.tTTL=150

(7) Add IPs from (6) to list of zbot

proxy network IPs

Zbot CnCs Monitoring System

Zeus

Config URLs

Binary URLs

Drop Zone URLs

Citadel

KINS

&

Ice IX

Asprox

Zemot/

Rerdom

Phishing

Ursnif

Madness Pro

Pony panel

newGOZ

Tiny

Banker

Malware phoning to CnC domains

Tiny Banker CnCs example

Tinba domains detected by FF

model

Get network reports for all associated known samples

Extract queried domains from network traffic

reports

Apply filtering heuristics to remove FPs (traffic pattern, lexical features, etc)

Fastflux Case Study: Zbot

• Collecting live intel helps learn about bad actors TTP

• Register domains with evasive names to confuse trackers

e.g. suspended-domains-nic.biz looks as a suspended domain,

in reality it’s a recent NS domain (Jan 14th) for zbot FF CnCs

• [a-d].suspended-domains-nic.biz

[dns1-dns4].suspended-domains-nic.biz

-> are authoritative name servers for zbot FF domains

The name servers are themselves hosted on the zbot proxy

network -> double flux set up

Registrar

r01-reg

TodayNic

r01-ruRegru-ru

Paknic

Melbourne IT

Registrar

Netlynx

Web Commerce

Ardis-reg

ru-center-ru

regru-reg

Rogue or abused registrars

http://spamtrackers.eu/wiki/index.php/R01.ru

http://spamtrackers.eu/wiki/index.php/R01.ru

EmailMX

RR

EmailMX

RR

No MX record

FakeMailGenerator

Picamail - Google 85Mail - Google

Privacy - TopDNS

GMX.com

HotmailYandex

DGA case study:

new GameOver Zeus (newGOZ)

newGOZ Background

What is a DGA?

Conficker 2008

Typically calculated on time/day/date

Letter based vs dictionary based

Gameover Zeus “newGOZ”

letter based with salts to extend algorithm (2

known)

11000 possible domains per day

Oct 7 – Dec 7 (62 days)

newGOZ Tracking System

OverviewIdentify a DGA VirusTotal, TotalHash, Intel sharing

communities

Query patterns: cooccurences, spikes,

lexical analysis

Reverse DGA algorithm Hexrays decompiler, IDA, Hopper,

Ollydbg

Predict Daily C2 Domains Python+BASH+massresolver

Yesterday, today, tomorrow (for overlaps)

682,000 possible C2 domains over 62 days

Oct 7 - Dec 7

Identify live C2 Domains Attempt to resolve domains every TTL

seconds (5 minutes)

251 resolved (evil and researchers)

Probe for information on C2 Domains Whois, DNS, IP, ASN info for C2 and

authoritative domains

Enrich probe information with passive data PassiveDNS, historic whois, IP reputation

newGOZ Domain TTLs

251 different C2 domains resolved

Domain Count TTL Alignment

110 300 Evil

81 10800 Sinkhole

58 666 Sinkhole

9 3600 Sinkhole

5 1800 Sinkhole

4 600 ?

1 7200 ?

1 14400 ?

Domain with multiple TTLs changed owners

newGOZ C2 Name Servers

31 authoritative domains (2LD)

21 name servers had ns1 and ns2 pairs

5 domains (likely more) are researchers’

4 name servers were eventually parked

possibly due to not resolving

possibly due to not existing


a.dns.gandi.net

b.dns.gandi.net

c.dns.gandi.net

dns1.registrar-servers.com





ns.123-reg.co.uk

ns2.123-reg.co.uk

ns01.domaincontrol.com


pdns05.domaincontrol.com


ns1.torpig-sinkhole.org


ns1.sinkhole.ch

ns2.sinkhole.ch

ns1.dynadot.com

ns2.dynadot.com

ns1.ilcriminallaw.net.lamedelegation.org

ns1.acutica.net.rcom-dns.eu

ns1.ezracesite.net.rcom-dns.eu

ns1.the-jumbotron.net.rcom-dns.eu

ns1.acutica.net

ns1.autozphibsnz.com

ns1.bethanychildcare.net

ns1.borrowbynet.net

ns1.bossvietguider.com

ns1.bundesligagame.net

ns1.energiazielona.net

ns1.ezracesite.net

ns1.hitzandronum.net

ns1.hotinspiritrees.net

ns1.ilcriminallaw.net

ns1.israelandpalestin.com

ns1.longhilpartners.com

ns1.lovecapo.net

ns1.overbytes.net

ns1.rannfyaether.net

ns1.the-jumbotron.net

ns1.themobpokershop.net

ns1.thepurringpiano.net

ns1.videohomebing.com

ns1.visiteitacares.com

ns1.whiterelicons.com

ns1.zoogmusics.net

ns1.zumbbawecker.net

-



ns2.borrowbynet.net


-


-

-

-

-

-


ns2.lovecapo.net

ns2.overbytes.net


-



-


-

ns2.zoogmusics.net

-


a.dns.gandi.net

b.dns.gandi.net

c.dns.gandi.net






ns.123-reg.co.uk

ns2.123-reg.co.uk







ns1.sinkhole.ch

ns2.sinkhole.ch

ns1.dynadot.com

ns2.dynadot.com

ns1.ilcriminallaw.net.lamedelegation.org

ns1.acutica.net.rcom-dns.eu

ns1.ezracesite.net.rcom-dns.eu

ns1.the-jumbotron.net.rcom-dns.eu

ns1.acutica.net



ns1.borrowbynet.net


ns1.bundesligagame.net


ns1.ezracesite.net

ns1.hitzandronum.net

ns1.hotinspiritrees.net

ns1.ilcriminallaw.net

ns1.israelandpalestin.com


ns1.lovecapo.net

ns1.overbytes.net


ns1.the-jumbotron.net



ns1.videohomebing.com


ns1.whiterelicons.com

ns1.zoogmusics.net

ns1.zumbbawecker.net

-



ns2.borrowbynet.net


-


-

-

-

-

-


ns2.lovecapo.net

ns2.overbytes.net


-



-


-

ns2.zoogmusics.net

-

Researchers

Parked

Evil

Evil NS1 Evil NS2

newGOZ C2 Domain Registrars

Dynadot

GoDaddy

1&1 Internet AG

101Domain

Bigrock Solutions

Enom

Gandi SAS

Melbourne IT DBA Internet Names Worldwide

Network Solutions

TodayNIC

Turncommerce DBA NameBright

Webfusion

Registrar

Registrar

1&1 Internet AG

Dynadot

Gandi

TodayNic

Melbourne IT

Bigrock

SolutionsTurnCommerce

DBA

NamebrightGoDaddy

101Domain

Enom Webfusion

Network

Solutions

newGOZ Registrant Email Addresses

99 different registrant emails (C2 and NS domains)

NOT including confirmed researchers

Some accounts were created, some weren’t

[email protected]

[email protected]

[email protected]

[email protected]

NameBright

Privacy

TodayNic Privacy

(No MX RR)

YahooGMX.com

AOL

Enom Privacy

(whoisguard)

GoDaddy

Privacy(Domainsbyproxy)

GMX.netHotmail

Zoho

FakeMailGenerator

newGOZ C2 and NS Hosting

86 C2 and NS IPs

54 unique hosting locations

3 providers used by known researchers

Mix of VPS, ISP, and compromised

12 Amazon

8 GoDaddy

4 GANDI SAS

3 Rackspace Hosting

3 OVH

3 Confluence Networks Inc

3 1&1 Internet AG

2 Webfusion Internet Solutions

2 ViaWest

2 SoftLayer Technologies Inc.

2 PT Jastrindo Dinamika

2 Black Lotus Communications

1 Yuli Azarch trading as YaiSales

1 XL Internet Services B.V.

1 Viet Solutions Services Trading Company Limited

1 Viasat Communications Inc.

1 VDSINA VDS Hosting

1 TTNETDC Turkiye Telekom Data Center

1 TANET-BNETA, Taiwan

1 Symphony Communication Plc

1 SPARK NEW ZEALAND TRADING LIMITED

1 Shandong technology university

1 Rook Media USA, Inc.

1 RIPE Sinkhole

1 RCS & RDS Business

1 Radore Veri Merkezi Hizmetleri A.S.

1 NOS COMUNICACOES S.A. (TVCABO-

Portugal)

1 Namecheap, Inc.

1 MonsterCommerce, LLC

1 Ministry of Education Computer Center, Taiwan

1 Ministère de l'aménagement du territoire de

l'équipement et des transports

1 Kornet - Korea Telecom

1 KMS-Hosting.com Customers

1 Kabel Baden-Wuerttemberg GmbH & Co. KG

1 Joe's Datacenter, LLC

1 Indiana University

1 ID Uppal Private Limited

1 HOST1FREE.COM VPS services

1 HONGIK UNIVERSITY

1 HANANET - broadNnet

1 Google Cloud

1 GHOSTnet Network used for VPS Hosting

Services

1 Gelderland Internet Exchange - Dedicated Servers

1 FortaTrust USA Corporation

1 EXMOS-LIMITED

1 ERX-NETBLOCK

1 CloudFlare, Inc.

1 Cizgi Telekom

1 China Mobile communications corporation

1 Bharti Tele-Ventures Limited

1 Belgacom ISP SKYNET-CUSTOMERS

1 Argon Data Communication

12 Amazon

8 GoDaddy

4 GANDI SAS

3 Rackspace Hosting

3 OVH

3 Confluence Networks Inc

3 1&1 Internet AG

2 Webfusion Internet Solutions

2 ViaWest

2 SoftLayer Technologies Inc.

2 PT Jastrindo Dinamika

2 Black Lotus Communications

1 Yuli Azarch trading as YaiSales

1 XL Internet Services B.V.

1 Viet Solutions Services Trading Company Limited

1 Viasat Communications Inc.

1 VDSINA VDS Hosting

1 TTNETDC Turkiye Telekom Data Center

1 TANET-BNETA, Taiwan

1 Symphony Communication Plc

1 SPARK NEW ZEALAND TRADING LIMITED

1 Shandong technology university

1 Rook Media USA, Inc.

1 RIPE Sinkhole

1 RCS & RDS Business

1 Radore Veri Merkezi Hizmetleri A.S.

1 NOS COMUNICACOES S.A. (TVCABO-

Portugal)

1 Namecheap, Inc.

1 MonsterCommerce, LLC

1 Ministry of Education Computer Center, Taiwan

1 Ministère de l'aménagement du territoire de

l'équipement et des transports

1 Kornet - Korea Telecom

1 KMS-Hosting.com Customers

1 Kabel Baden-Wuerttemberg GmbH & Co. KG

1 Joe's Datacenter, LLC

1 Indiana University

1 ID Uppal Private Limited

1 HOST1FREE.COM VPS services

1 HONGIK UNIVERSITY

1 HANANET - broadNnet

1 Google Cloud

1 GHOSTnet Network used for VPS Hosting

Services

1 Gelderland Internet Exchange - Dedicated Servers

1 FortaTrust USA Corporation

1 EXMOS-LIMITED

1 ERX-NETBLOCK

1 CloudFlare, Inc.

1 Cizgi Telekom

1 China Mobile communications corporation

1 Bharti Tele-Ventures Limited

1 Belgacom ISP SKYNET-CUSTOMERS

1 Argon Data Communication

NS IP Address C2 DomainIP Address

Malware Cabal

Sinkhole

VirusTracker

Sinkhole

???

Godaddy

Arbor Networks

Sinkhole

???

Godaddy

Badness

NS IP Address C2 DomainIP Address

newGOZ Now

No new evil domains registered since 12 Nov 14

why?

speculation:

not resilient without peer-to-peer

abandoned for new malware

silent LE take down

Sinkholes are still active

oldGOZ Client Queries

oldGOZ generates 1000 domains every 7 days starting

from first of the month (except 1st and last batch)

Dec 1 - Dec 6 Jan 1 – Jan 6

Dec 7 - Dec 13 Jan 7 – Jan 13

Dec 14 – Dec 20 Jan 14 – Jan 20

Dec 21 – Dec 27 Jan 21 – Jan 27

Dec 28 – Dec 31 Jan 28 – Jan 31



newGOZ Client Queries (to add)

newGOZ generates 1000 domains/day using one of the salts

10,000 domains/day using the other salt

newGOZ Take Aways

Important things to note about newGOZ infrastructure

TTLs of domain names (300)

Use round-robin DNS (multiple IPs per domain)

Registrar preferences (TodayNic, Melbourne IT,

BigRock)

Registration to resolution delta (~1 day)

Registrant email pattern

Many C2 IPs, many NS IPs

Use of compromised (and possibly dedicated) IPs

newGOZ tracker:

Snapshooter

newGOZ Improved Tracking

SystemJSON instead of flat text output

Pure Python instead of BASH, Python and C

Client

generates GOZ domains

identifies resolving domains

maps resolving domains to workers

spawns a dedicated client process for each worker

asynchronously sends requests to workers

Workers

daemon waiting for client tasks requests

queries the DNS, whois, etc.

GOZ DGA

p1 p2 p3

Client

workerd

NS RRs

whois

server

whois

server

whois

server

workerd

workerd

NS NS

8.8.8.

8IP RR

COUNT=0;

while [ ${COUNT} -lt 20 ];

do dig +short whois.verisign-grs.com;

COUNT=$[${COUNT}+1];

sleep 1;

done | sort | uniq -c

5 199.7.48.74

4 199.7.50.74

11 199.7.56.74

newGOZ Snapshooter Demo

github.com/anthonykasza/snapshooter

Snapshooter: ToDo

- Automatically contact registrars and hosting

providers with complaints

- Collect content hosted on domain

- Graph database backend

- Pray for RDAP drafthttps://tools.ietf.org/html/draft-ietf-weirds-json-response-10

Conclusion

• Threat Intelligence is crucial to make strategic &

tactical decisions for reactive & proactive security

• Different techniques to collect network threat intel.

– Active probing

– Passive Monitoring

• Fastflux: Zbot fast flux proxy network

• DGA: GameOver Zeus botnet

• Snapshooter

References

-Catching malware en masse: DNS & IP style, D. Mahjoub,

T. Reuille, A, Toonk, BlackHat 2014, DefCon 2014

-Sweeping the IP space: The Hunt for Evil on the Internet,

D. Mahjoub, Virus Bulletin 2014

-A New Look at Fast Flux Proxy Networks, D. Mahjoub, H.

Adrian, BotConf 2014

-DNS Analytics, O. Kamal, BotConf 2014

-ZeuS Tracker

-Massresolver, F. Denis, github.com/jedisct1/massresolver

-http://www.malware-traffic-analysis.net/

Acknowledgements

OpenDNS

ShmooCon

Arbor Networks (initial newGOZ DGA)

John Bambenek

Thank You.

Questions?

@dhialite

@anthonykasza

Technology

Infrastructure Tracking with Passive Monitoring and Active Probing: ShmooCon 2015 Presentation