DNSSEC: Signing, Validating, and Troubleshooting · Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science DNSSEC: Signing, Validating, and Troubleshooting

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

DNSSEC: Signing, Validating, and Troubleshooting

Michael Sinatra

Energy Sciences Network

Internet2/ESCC Joint Techs

Summer 2012


What this tutorial is about

•  Understanding why DNSSEC is important

•  Learning the major concepts of DNSSEC

•  Understanding how to sign and validate DNSSEC records

•  Learning how to use advanced tools to troubleshoot DNS and DNSSEC issues.

•  dig •  dnscap/nmsg •  others

•  Understanding DNS and DNSSEC pitfalls and where you can get into trouble.

•  Learning how basic troubleshooting techniques can help you solve complex problems.


What this tutorial is not about

•  An introduction to DNS.

•  A commentary on the “complexity” of DNSSEC and why we should throw up our arms and run away (we definitely shouldn’t).

•  A discussion of other topics and implementations, such as DNScurve or DNS-over-SSL or –IPSEC.

•  I assume a working knowledge of DNS (although I do provide some basic concepts and definitions), and also of UNIX, its command line, and how to install and/or compile packages in your favorite modern UNIX variant.


Why DNSSEC?

•  DNS is basically asecure. •  The asecurity of DNS creates an insecurity for Internet applications

•  DNSSEC fixes a big chunk of the problem •  Provides integrity of the data—we can know that it wasn’t altered

in transit •  Provides authenticity—we can know that it came from the real

domain owner •  We can “know” these things based on our trust of the cryptography

and the competency of the domain owner! •  Once the DNS is (more) secure, there are other cool things you

can do with it! •  DNSSEC does not provide confidentiality

•  DNS remains a public database and should be treated as such


DNSSEC timeline (stolen from Matt Larson)

•  1993: Discussion of secure DNS begins

•  1994: First draft of possible standard published

•  1997: RFC 2065 published (DNSSEC is an IETF standard)

•  1999: RFC 2535 published (DNSSEC standard is revised)

•  2005: Total rewrite of standards published •  RFC 4033 (Introduction and Requirements) •  RFC 4034 (New Resource Records) •  RFC 4035 (Protocol Changes)

•  July 8, 2008: Kaminsky Vulnerability disclosed


DNSSEC timeline (stolen from Matt Larson)

•  July 15, 2010: Root zone signed

•  July 29, 2010: .edu signed

•  December 9, 2010: .net signed

•  March 31, 2011: .com signed

•  à From Matt Larson, DNSSEC Overview, NANOG 53 & 54.

•  Why was Kaminsky so important? •  Showed that the small query ID field was TOO small. •  But djb said the same thing in 1999. •  Kaminsky figured out how to exploit the small query ID field by

forcing queries for sibling records and spoofing responses with in-bailiwick ADDITIONAL sections.


Basic DNS overview - concepts

•  Domain •  Zone: Contains the DNS records for a domain and possibly for child

domains. •  Parent domain/zone, child domain/zone •  Zone cut, top of zone

•  DNS Resource Record (RR): The basic DNS record within a zone •  name (rarely called LDATA) •  type (A, AAAA, NS, MX, etc.) •  RDATA: The target of the record; for a A or AAAA record, it’s an IP

address; for an MX record, it’s a hostname. •  TTL: Time to live – the length of time the particular RR should be

cached in a recursive DNS server



•  Authoritative DNS server (or Authority): A server that holds the authentic data for a particular domain or domains.

•  Recursive/caching DNS server: A server that the client queries for Internet DNS names and records. The recursive server then, in turn, queries various authoritative servers to get information about the requested DNS record.



•  Previous slide: https://en.wikipedia.org/wiki/File:An_example_of_theoretical_DNS_recursion.svg



!

!

!

!

es.net. ! ! !600 !IN!A !128.55.22.201!

!

Name

TTL

Type

RDATA



!

!

!

!

es.net. ! ! !600 !IN!A !128.55.22.201!

!

The whole thing is the RR.



•  An RRSET consists of all of the RRs with the same name and type. !

es.net. ! ! !600 !IN!A !128.55.22.201!

!

!

!

es.net. ! ! !172800 !IN!NS!ns-lvk.es.net.!

es.net. ! ! !172800 !IN!NS!ns-aoa.es.net.!

es.net. ! ! !172800 !IN!NS!ns1.es.net.!

!

!



•  An RRSET consists of all of the RRs with the same name and type. •  The concept of the RRSET is important because DNSSEC signs

each RRSET, not each RR. •  This is done with the RRSIG record:

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!

!! ! !ebvVfhe5hcZ5Uirg/U78WRmFO7ohq5HOJpTEhkVf9OMe!

!! ! !Tz6JkRIOu10aBi3p+ReEQtppjNEpViKenkc6w+8pTS0T!

!! ! !73S1iTMDZUZn3b2INclqq3kZtC//6ULQYQywwso7cGJE!

!! ! !nn0i+/3v70pMfHbs/+W2/vmnuwR/zAqYehfKNws= )!

!

!


Basic DNS/DNSSEC overview - concepts

•  RRSIG records (i.e. RRs with a type of RRSIG) contain the cryptographic signature for a particular RRSET (noted in the RDATA of the RRSIG record).

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!





!

!



!

!

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!





!

!

RRSET being signed



!

!

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!





!

!

RRSET being signed Algorithm



!

!

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!





!

!


TTL of original RRSET



!

!

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!





!

!



Sig expiration



!

!

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!





!

!



Sig expiration

Sig valid date



!

!

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!





!

!



Sig expiration

Sig valid date



!

!

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!





!

!



Sig expiration

Sig valid date



!

!

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!





!

!ID of signing key



!

!

es.net. ! ! !75656 IN RRSIG NS 5 2 172800 (!

!! ! !20120715100056 20120701090056 1695 es.net.!





!

!Labels field: Shows how many labels are in the name field



•  RRSIGs are important, but there are other critical DNSSEC records: •  DNSKEY: Contains the public key of the keypair that is being used

to sign the DNS record(s). •  NSEC/NSEC3: Used to provide proof of non-existence of DNS

records in a DNSSEC-signed zone. •  DS: Delegation Signer – used to establish a chain of trust between

the parent and child zone.


Extremely basic cryptography

•  Public key? Keypair? What’s that and what does it have to do with DNSSEC?

•  It’s important to understand some basic cryptography concepts. •  Symmetric-key cryptography: A single* key or shared secret is used

to both encrypt and decrypt a message. Examples: AES, DES, Blowfish, RC5.

•  Asymmetric-key cryptography (aka public-key cryptography): Defines a keypair such that a message encrypted by the private key can be decrypted by the public key and possibly vice versa, but no single key can both encrypt and decrypt a message. Examples: RSA, DSA, Diffie-Hellman, ECDSA.

•  One-way cryptographic hash or digest: Produces a unique and consistent string of bits from a message. Changes in the message will change the string, and the message cannot be decrypted from the string. Examples: MD5, SHA-1, SHA-2, SHA-3.



•  What’s all this about encryption? We’re not encrypting DNS records, are we?

•  No, but public key cryptography provides a way to digitally sign a message, such as a DNS record.

•  Kindergarten explanation: •  Take a one-way hash (say, SHA-1) of the DNS RRSET. •  Encrypt that hash with your private key. •  The resulting signature goes into the RRSIG. •  To validate the signature, decrypt the hash with the public key of

the signer. •  Take a one-way hash of the same DNS record, and compare it to

the hash you just decrypted from the signature.



•  The Wikipedia version of the previous slide: •  http://en.wikipedia.org/wiki/File:Digital_Signature_diagram.svg



Potential weaknesses

•  Cryptography (particularly public-key cryptography) relies on the relative infeasibility but not impossibility of performing certain mathematical functions, e.g., factoring large integers.

•  RSA, for example, calls for generating two large prime numbers at random whose product is reallyN hard to factor.

•  But if your randomization function is weak and/or predictable, factoring gets a whole lot easier.

•  Similarly hash functions can suffer from collision weaknesses, where two different messages produce the same hash! MD5 has known collisions that are being exploited. SHA-1 has theoretical collisions, but they have been difficult to exploit. Not clear how easy it is to exploit these collisions in a DNSSEC context.

•  Think about this when you pick your signing algorithm!


Time for a quiz!

busteddns.com. 86400 IN SOA ns.busteddns.com. hostmaster.busteddns.com. (!

1 ; serial!

10800 ; refresh (3 hours)!

900 ; retry (15 minutes)!

604800 ; expire (1 week)!

900 ; minimum (1 day)!

)!

IN NS ns1.busteddns.net.!

IN NS ns2.busteddns.net.!

IN A 192.0.2.123!

!! ! ! IN ! AAAA 2001:db8::2!

www IN AAAA 2001:db8::1:2!


DNSSEC records (continued)

•  NSEC and NSEC3: •  Used to prove non-existence of records. •  Shows all RRSETs that exist for a given name and also shows the

next record in canonical order. By signing the NSEC record, we “prove” that there are no other records between the two DNS names.

berkeley.edu.! !300 !IN!NSEC !10000.berkeley.edu. A NS SOA MX TXT AFSDB AAAA RRSIG NSEC DNSKEY!

•  Why might some people be concerned about this record? -  Enumeration -  Issues with very large delegation zones (e.g. .com)



•  NSEC and NSEC3: •  NSEC3 solves this problem. Hashes the names and can be

configured to only be inserted for secure delegations (i.e. opt-out). !

A1RT98BS5QGC9NFI51S9HCI47ULJG6JH.net. 86400 IN NSEC3 1 1 0 - A2003PRAPCHMS9L1A11GMVJ0JNP84A46 NS SOA RRSIG DNSKEY NSEC3PARAM!

Hash of DNS name






Hash of “next” name






NSEC3 parameters (also in NSEC3PARAM record)



•  The DS record: Delegation Signer is a digest (i.e. hash) of a child zone’s public key. The DS record is signed by the parent in order to establish a chain of trust between the parent and the child.

•  e.g.: •  . [root] has a signed DS record matching net’s public key. •  net has a DS record matching brokendns.net’s public key. •  brokendns.net has a DS record matching foo.brokendns.net’s

public key. •  à If you have a trust anchor for the root zone (i.e. you have a

trusted copy of root’s public key), then you can establish a chain of trust all the way down to foo.brokendns.net.

•  We’ll talk about trust later…



•  The all-important DNSKEY record. •  Contains a public key that matches one of the keys to sign the

zone. •  Not actually required to be in a signed zone (RFC4034 specifies its

inclusion as a SHOULD, not a MUST), but most signing tools won’t let you sign the zone without (automatically or manually) including the DNSKEY record.

•  Comes in two flavors: SEP (KSK) and non-SEP (ZSK). •  The standard specifies the SEP bit as the designation of a key

that has a DS record in the parent or a trust anchor somewhere else.

•  KSK and ZSK are not in any RFC; they are part of the general operational practice of DNSSEC. (Yes, that’s confusing.)



•  ZSK vs. KSK •  This distinction can make DNSSEC seem complex…but it’s not

really. •  KSK: Signs only the DNSKEY RRSET. To maintain a chain of trust,

KSK has the SEP bit set and matches the DS record in the parent. •  ZSK: Signs every record in the zone (may or may not sign the

DNSKEY RRSET). •  Actually, you don’t need both keys—you could sign your entire

zone with just one key.

•  Why might you want them both?



•  Example KSK for net. !!net. ! ! !82123 IN DNSKEY !257 3 8 (!!! ! !AQOYBnzqWXIEj6mlgXg4LWC0HP2n8eK8XqgHlmJ/69iu!!! ! !IHsa1TrHDG6TcOra/pyeGKwH0nKZhTmXSuUFGh9BCNiw!!! ! !VDuyyb6OBGy2Nte9Kr8NwWg4q+zhSoOf4D+gC9dEzg0y!!! ! !FdwT0DKEvmNPt0K4jbQDS4Yimb+uPKuF6yieWWrPYYCr!!! ! !v8C9KC8JMze2uT6NuWBfsl2fDUoV4l65qMww06D7n+p7!!! ! !RbdwWkAZ0fA63mXVXBZF6kpDtsYD7SUB9jhhfLQE/r85!!! ! !bvg3FaSs5Wi2BaqN06SzGWI1DHu7axthIOeHwg00zxlh!!! ! !TpoYCH0ldoQz+S65zWYi/fRJiyLSBb6JZOvn!!! ! !) ; KSK; alg = RSASHA256; key id = 35886!

Key flags




Signifies a DNSSEC key




Algorithm




Note that newer versions of dig (w/+multi option) decode all of this stuff for you (plus key id)!



•  Example ZSK for net. !!net. ! ! !82123 IN DNSKEY !256 3 8 (!!! ! !AQPb+N5c8KRQoU9UpaQM19EyjUUwWME+FmddI0tV5CPK!!! ! !Dk0Kl3xJhiQuFTb2Uac5PKJw7Pc0U6fxf5g6UUE1kQ28!!! ! !vsn0ztk4gREdqXbWVwxOfT6XS3eUhk9Yaf55yEWn8sA+!!! ! !o9n17Ec3Y57i7mZqeC5QRzdOmOJG/ChR+juHMMNDrw==!!! ! !) ; ZSK; alg = RSASHA256; key id = 34268!

!•  The only major differences are that the key flag is 256 (==SEP bit

not set==ZSK) and the key length is shorter (hint! this is one of the reasons to have two keys…).

•  Note that in the net zone, only the KSK signs the DNSKEY RRSET. Let’s check some other zones and see if that’s the case.



•  Commonly used signing algorithms:

No. Algorithm Remarks 1 RSAMD5 Actually never used; deprecated 3 DSA Original DSA always uses SHA1 5 RSASHA1 NSEC only 6 DSA NSEC3, with SHA1 hash 7 RSASHA1 NSEC3 8 RSASHA256 NSEC or NSEC3 10 RSASHA512 NSEC or NSEC3 12 ECCGOST 256-bit hash ECC algorithm 13 ECDSAP256SHA256 ECC P-256 with SHA256 14 ECDSAP384SHA384 ECC P-384 with SHA384


DNSSEC concepts: Keys & Algorithms

•  Algorithm rollovers: Changing algorithms in a zone that is already doing DNSSEC in production (i.e. is signed and has DS records in the parent) is REALLY tricky.

•  It’s a good idea to decide on the right algorithm from the beginning.

•  Many are currently using RSASHA256: combines decent performance with encryption strength.

•  But that may not be the only reason to choose something. •  Computers getting faster so RSASHA512 might be okay. •  And what about those ECDSA algorithms?



•  ZSK rollovers are easy. You can choose to increase the bits of the same algorithm (e.g. RSA 1024 à RSA 2048) by just doing a key rollover.

•  KSKs not as easy, but easier than algorithm rollovers.

•  Why roll keys? •  Possibility of factoring. •  With some algorithms, keys may be easier to factor the more they

are used. Or, there may be more risk if they are used to sign a zone where a weak random number generator is used.

•  Increase key strength within an algorithm. •  Rumsfeld’s Razor.

•  Remember: Private keys need to be kept secure!



•  ZSK rollovers are easy. They can be automated, and typically are. •  Usually pre-publish new key, but don’t sign with it. •  Then begin signing with new key, stop signing with the old key, but

don’t remove the old key yet. •  Then remove the old key. •  à For each of the above steps, wait (1 or 2) x (the longest TTL for

RRSIGs in the zone) between steps. Most automated tools will also have a notion of a “propagation delay,” which is the time it takes for the zone to get pushed out to all authoritative DNS servers.



•  KSK rollovers are a bit trickier. You can either: •  Pre-publish the DS record in the parent, and then use the same

pre-publish method as used in the ZSK rollover. •  Double-sign:

•  Generate new KSK and begin signing with it as well as the old key.

•  Request replacement of the DS record in the parent. •  Remove old KSK. •  à Again, you would want to wait until all TTLs (including negative

TTLs) have expired between each steps. •  Note that you can also double-sign ZSK rollovers, but that will

make your zones BIG!


DNSSEC concepts: Trust

•  We now have an understanding of the key (no pun intended) components of DNSSEC. How do we use these components to build a chain of trust?

•  Trust models for various public-key security schemes:

•  Note that DNSSEC does not use certificates!

Security scheme Trust model PKI/SSL/TLS Hierarchical, multi-root (or internal) PGP (inc. GPG) Anarchical (web-of-trust) DNSSEC Hierarchical, single-root (or internal) IPSEC All of the above and none (opportunistic) SSH Bi-lateral between hosts, or hierarchical

using DNSSEC!



•  DNSSEC’s trust model is rigid and hierarchical. It flows from the parent to the child zone.

•  You must trust someone at some point in the hierarchy. This involves configuring a “trust anchor”—the public key of the DNS zone where you wish to begin your trust.

•  à In most cases, this is the root zone.

•  It is also possible to have “islands of trust,” where you maintain trust anchors for zones that do not have a hierarchical chain of trust.

•  Internal zones •  Testing/experimentation •  Zones where I want my own clients to validate, but I don’t want the

general public to validate.



•  Alternative trust methods: DLV •  I already discussed private trust anchors, and there’s another

method (other than the standard hierarchical root trust anchor): Domain Lookaside Validation (DLV).

•  Provides an alternative key repository infrastructure. •  Instead of starting at the root and building a chain of trust

downward, starts with the domain in question and works upward toward the root.

•  Uses DLV record.



•  DLV record:

burnttofu.net.dlv.isc.org. 3575 !IN!DLV !804 5 1 43EFC4842630C4AEA45C63AC4C0A861B45650650!

burnttofu.net.dlv.isc.org. 3575 !IN!DLV !804 5 2 D24944A8D2F8363AAD02BE83E04CCF217388DB22BD3400C2A929ED6C 1CFE9243

•  Looks like a DS record, doesn’t it?

•  dlv.isc.org is a popular DLV run by ISC, but it isn’t the only one. SecSpider is another.



•  DLV reverse hierarchy: •  I want to try to validate foo.burnttofu.net:

•  Check if there is a DLV record for foo.burnttofu.net.<my-favorite-DLV-registry> (e.g. foo.burnttofu.net.dlv.isc.org).

•  If not, check if there is a for burnttofu.net.<my-favorite-DLV-registry> and then look for DS record chain.

•  If not, check if there is a for net.<my-favorite-DLV-registry> and then look for DS record chain.

•  If not, then there is no trust anchor or chain of trust. Zone is insecure (i.e. not DNSSEC-protected, but not invalid).



•  DLV was originally intended as a stop-gap measure to provide a chain of trust for zones that might otherwise be orphaned by gaps in the “regular” chain of trust.

•  This was especially true before the root and the major TLDs were signed.

•  But it is still useful. Why? •  Zones within an organization where the organization isn’t signing

its zone. •  Zones that are part of a TLD hierarchy that isn’t signed. •  Zones that use a registrar that doesn’t yet support the insertion of

DS records. (Dyn, Godaddy, and a few others currently support DS records. Last I checked, Network Solutions didn’t and required a special email to their “DS record guy” to update the DS record.)



•  DLV may be useful as an alternative key distribution method, alongside the existing root-based hierarchical trust method.

•  Could guard against botched root key rollovers—currently a single point of failure.

•  Some modifications could/would need to be made to DLV.

•  This is my idea and it is, let’s say, “really controversial.”

•  Just sayin’.


Decisions, decisions

•  What algorithm to use? •  Use both KSK and ZSK? •  Whether to publish your public key digest to establish a chain of

trust? •  How to publish your public key digest to establish a chain of trust

(e.g. DS in parent or DLV)? •  If DLV, which DLV to use?

•  How often/whether to roll KSKs and/or ZSKs? •  How long to make signature intervals? (We’ll talk about that

later…) •  Which tools to use? •  To automate or not to automate (whoops, that’s not really a decision

—you MUST automate!)


Decisions, decisions

•  You can sign zones manually. There are two major toolsets that that do the work: BIND and LDNS.

•  BIND: This is the BIND that you know and love. The DNSSEC signing tools are included with the BIND distribution.

•  LDNS: Written by NLNetLabs as an alternative to the BIND codebase—same as nsd and unbound. Friendly competition to ensure a diverse codebase.

•  In addition to the two that I will discuss, PowerDNS also has full DNSSEC support and can even do inline-signing! However, I am not as familiar with PowerDNS, and can’t talk too much about it. The omission of information about PowerDNS and other tools is a limitation of this tutorial and its author, not of the omitted tools.

•  Let’s sign some zones.


DNSSEC Tools

•  First we need to create some keys. Think about what algorithm you want to use. Feel free to play along if you have a Mac or Un*x system with BIND installed.

•  BIND tools: •  ZSK:

dnsseckeygen a RSASHA256 b 1024 n ZONE \ busteddns.com •  KSK:

dnsseckeygen a RSASHA256 b 2048 n ZONE -f KSK \ busteddns.com!

!

Algorithm


DNSSEC Tools





!

Key length in bits


DNSSEC Tools





!

This means we’re generating a DNSSEC key


DNSSEC Tools





!

-f KSK means we’re generating a key with the SEP bit set; otherwise it’s a ZSK


DNSSEC Tools





!

The zone for which the key is being generated


DNSSEC Tools

•  A simple command for signing a zone: •  BIND tools: dnssec-signzone -o busteddns.com. -e +518400 zone.db K*.private!

How long before the RRSIGs expire


DNSSEC Tools


Actual filename(s) for keys with which to sign the zone


DNSSEC Tools


•  Note that this command won’t work unless you manually add the public keys to the zone as DNSKEY records! But here’s a shortcut:

dnssec-signzone -o busteddns.com. -S -e +518400 zone.db K*.private!

Smart signing: Automagically include correct DNSKEY records!


DNSSEC Tools

•  A simple command for signing a zone: •  LDNS tools: ldns-signzone -o busteddns.com zone.db Kbusteddns.com.+008+50865!

Note that LDNS wants the key name, not the filename, for each key! BUT…it inserts the DNSKEY records automagically!


DNSSEC Tools

•  That’s it! You now have a signed zonefile (zone.db.signed), which you can place in your BIND configuration:

zone “busteddns.com" {!

type master;!

file "master/busteddns.com/zone.db.signed";!

};!

!

•  Now, all you have to do is remember to log into your nameserver, run that command again, and reload your nameserver every time you make a change to the zone or when the signatures are about to expire. Uh, yeah, that doesn’t really scale.


DNSSEC Automation

•  This is why we keep saying that you MUST automate! So what are some of the ways that you can automate your signing?

•  Use BIND tools with more of the “smart signing” features (cron is your friend).

•  Use zkt, a nice wrapper around the BIND tools, but doesn’t take advantage of some of the newer features of the BIND tools (again cron is your friend).

•  Use a signing appliance like Secure64 or Infoblox. If you already use something like Infoblox for IPAM and/or DNS management, you might be able to “just turn DNSSEC on.” But make sure you understand what it’s doing.

•  Use BIND’s (or PowerDNS’s) auto-dnssec or inline-signing features.

•  Use OpenDNSSEC or DNSSEC-tools.


DNSSEC Automation

•  BIND “smart-signing” and inline-signing. •  Takes advantage of key metadata fields in later versions of BIND. •  Specify when keys are to be activated, published, deactivated, and

removed. •  After you have specified the parameters on the first key, you can do “dnssec-keygen -S <key>” to create a successor key with automatically-generated parameters! (Although you will need to manually set the inactivation date.

•  First, generate the key with proper parameters: dnssec-keygen -a RSASHA512 -b 1024 -P now -A now+23d -I now+60d -D now+120d busteddns.net!


DNSSEC Automation




Publication date


DNSSEC Automation




Activation date


DNSSEC Automation




Inactivation date


DNSSEC Automation




Deletion (from zone) date


DNSSEC Automation

•  BIND “smart-signing” and inline-signing. •  Now, generate a successor key: dnssec-keygen -S Kbusteddns.net.+010+22967!

•  Configure BIND correctly: zone "busteddns.net" {! type master;! inline-signing yes;! auto-dnssec maintain;! file "/etc/namedb/inline-signed/busteddns.net/

zone.db";! key-directory "/etc/namedb/inline-signed/

busteddns.net";! serial-update-method increment;!};!


DNSSEC Automation

•  Note that in this example, the key isn’t active yet. Let’s run through a sample, using BIND inline-signing, going from an unsigned zone to a fully signed and automated zone (with the next 2 ZSKs already generated for rolling). I promise to make mistakes!


DNSSEC Automation: Case studies

•  My own personal zones: Not very many of them, not very hierarchical, not a huge security requirement, hand-managed.

•  à BIND inline-signing

•  UC Berkeley: Very hierarchical, many zones (need to integrate DS records within UCB’s zone structure), not a huge security requirement, generated from a home-grown database application.

•  à zkt with a perl wrapper/glue script

•  ESnet: Not very hierarchical, not many zones, more significant security requirement, commercial IPAM which dynamically updates zones on a hidden master server.

•  à Inline signing using a Secure64 appliance



•  UC Berkeley: •  Zones are generated from a PostgreSQL database using a set of

home-grown applications and saved to disk on a staging server in BIND zone-file text format.

•  Zones are then pushed out via rdist (yes, rdist) to master DNS servers, where they are then offered to slaves via zone transfer.

•  Opportunity for staging server to sign zones after generating them from the database and before pushing them out.

•  Gotchas: •  Serial numbers will need to be updated properly. (Simple-dots

currently used.) •  Zone files only generated if there are changes. Signing tool will

need to re-sign zone if contents change or if the signatures need to be refreshed.



•  UC Berkeley: •  zkt will satisfy the requirements and can mitigate the gotchas, but it

creates some of its own gotchas: •  Assumes a directory structure (each zone in its own directory) that

was different from what was generated from the DB (flat forward and reverse directories with all zonefiles in them).

•  Needs a particular zonefile format in order to update the serial numbers. Works by updating the serial number in the source (unsigned) zone file.

•  Solution: Use a perl script to create the proper directory structure when zones are added, invoke named-compilezone to reformat the zone in the source directory and output it in the new directory structure whenever the zonefile in the “old” structure is newer than the zonefile in the “new” structure. Then run zkt.



•  ESnet: •  Uses Nixu NameSurfer as IPAM and DNS management. (One of the

first to support IPv6, it now supports DNSSEC, but did not at the time.)

•  Runs a master DNS server on the same host that runs NameSurfer; master server offers new zone every time updates are made.

•  Not easy to use BIND tools or zkt on the server, but ideal for a separate inline-signing appliance.

•  Secure64 also provides a built-in HSM for additional, FIPS-compliant security.

•  While UCB uses “bump-in-the-stack” signing, ESnet uses “bump-in-the-wire” signing.



•  Hopefully, these case studies can give you an idea as what kind of signing options are right for you.

•  As BIND tools become more sophisticated (e.g. key management daemons to go along with BIND inline-signing), it may become easier for both UCB and ESnet to go to BIND inline-signing.

•  Hardened BIND server could theoretically replace Secure64, although we would lose the HSM capability. Secure64 has numerous other hardening features as well.

•  Staging server (with mirrored backup) at UCB could turn into an inline-signed DNS server that would push zones via zone transfer and NOTIFY instead of rdist.

•  What do you think? What would work in your operation?


Validation

•  So far, we have talked about signing zones, which is done on the authoritative servers to provide cryptographic information that could be used to prove the integrity and authenticity of the data in that zone. It’s up to the recursive server to actually validate the cryptographic data and prove integrity and authenticity.

•  Why? •  Signing your zones doesn’t mean bupkis unless you and others are

validating. You need to do validation if you want DNSSEC to be worth anything.

•  Fortunately, turning on validation is easy. Some implementations even come with it on by default.


Validation

•  So far, we have talked about signing zones, which is done on the authoritative servers to provide cryptographic information that could be used to prove the integrity and authenticity of the data in that zone. It’s up to the recursive server to actually validate the cryptographic data and prove integrity and authenticity.

•  Why? •  Signing your zones doesn’t mean bupkis unless you and others are

validating. You need to do validation if you want DNSSEC to be worth anything.

•  Fortunately, turning on validation is easy. Some implementations even come with it on by default.


Validation

•  Validating recursive DNS servers: •  BIND •  unbound •  PowerDNS •  Microsoft DNS •  GbDns (Windows only, from George Barwood)


Validation

•  Turn it on in BIND:

options {!

!! ! […]!

!! dnssec-enable yes;!

dnssec-validation yes;!

managed-keys-directory "managed-keys";!

!! ! […]!

}!


Validation


options {!

!! ! […]!




!! ! […]!

}!

You don’t really need this in newer versions of BIND


Validation


options {!

!! ! […]!




!! ! […]!

}!

This is for RFC 5011-enabled zones. Trust-anchors will be updated as KSKs are rolled.


Validation

•  The managed-keys directory must be writable by BIND:

drwxr-xr-x 2 bind bind 512 Jul 8 09:05 managed-keys/!

•  Then you need to put the RFC 5011-managed zone’s trust anchor in named.conf under managed-keys {}:

managed-keys {! "." initial-key 257 3 8! "AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjF! FVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoX! bfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaD! X6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpz! W5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relS! Qageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulq! QxA+Uk1ihz0=";!};!


Validation

•  The managed-keys directory must be writable by BIND:

drwxr-xr-x 2 bind bind 512 Jul 8 09:05 managed-keys/!

•  Then you need to put the RFC 5011-managed zone’s trust anchor in named.conf under managed-keys {}:

managed-keys {! "." initial-key 257 3 8! "AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjF! FVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoX! bfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaD! X6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpz! W5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relS! Qageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulq! QxA+Uk1ihz0=";!};!

“initial-key” is the CURRENT public key for root at time of config. From now on it will be managed auto-matically as specified by RFC 5011.


Validation

•  Eventually, the managed-keys directory will get populated with files:

-rw-r--r-- 1 bind bind 720 Jul 8 09:05 cded85b10b1c5f620f4c05b28644400043ff056fa6682bd16e5c4fbf181d4b2f.mkeys!

-rw-r--r-- 1 bind bind 512 Jul 8 09:05 cded85b10b1c5f620f4c05b28644400043ff056fa6682bd16e5c4fbf181d4b2f.mkeys.jnl!

-rw-r--r-- 1 bind bind 645 May 21 2011 db-b5X1nVCj!

-rw-r--r-- 1 bind bind 720 Mar 7 09:03 managed-keys.bind!

•  managed-keys.bind has the current public key for the zone, so that BIND will know what it is when it restarts.!


Validation

•  You can run DLV at the same time:

options {!

!! ! […]!



dnssec-lookaside . trust-anchor dlv.isc.org;!


!! ! […]!

}!


Validation

•  Then include the trusted-keys section in the named.conf:

trusted-keys {!

!! ! dlv.isc.org. 257 3 5 "BEAAAAPHMu/5onzrEE7z1egmhg/WPO0+juoZrW3euWEn4MxDCE1+lLy2 brhQv5rN32RKtMzX6Mj70jdzeND4XknW58dnJNPCxn8+jAGl2FZLK8t+ 1uq4W+nnA3qO2+DL+k6BD4mewMLbIYFwe0PG73Te9fZ2kJb56dhgMde5 ymX4BI/oQ+cAK50/xvJv00Frf8kw6ucMTwFlgPe+jnGxPPEmHAte/URk Y62ZfkLoBAADLHQ9IrS2tryAe7mbBZVcOwIeU/Rw/mRx/vwwMCTgNboM QKtUdvNXDrYJDSHZws3xiRXF1Rf+al9UmZfSav/4NWLKjHzpT59k/VSt TDN0YUuWrBNh";!

};!


Validation

•  Let’s do the same with unbound. DNSSEC validation is already enabled in unbound, so all you need to do is add a trust-anchor:

server:!

![…]!

trust-anchor: ". DS 19036 8 2 49AAC11D7B6F6446702E54A1607371607A1A41855200FD2CE1CDDE32F24E8FB5”!

•  Note that the trust-anchor can be in DS format (or regular DNSKEY format).


Validation

•  To add DLV functionality, specify a DLV trust anchor (here it is placed in a separate file):

server:!

![…] !

!dlv-anchor-file: "dlv.isc.org.key” !

$ cat dlv.isc.org.key !

dlv.isc.org. IN DNSKEY 257 3 5 BEAAAAPHMu/5onzrEE7z1egmhg/WPO0+juoZrW3euWEn4MxDCE1+lLy2 brhQv5rN32RKtMzX6Mj70jdzeND4XknW58dnJNPCxn8+jAGl2FZLK8t+ 1uq4W+nnA3qO2+DL+k6BD4mewMLbIYFwe0PG73Te9fZ2kJb56dhgMde5 ymX4BI/oQ+cAK50/xvJv00Frf8kw6ucMTwFlgPe+jnGxPPEmHAte/URk Y62ZfkLoBAADLHQ9IrS2tryAe7mbBZVcOwIeU/Rw/mRx/vwwMCTgNboM QKtUdvNXDrYJDSHZws3xiRXF1Rf+al9UmZfSav/4NWLKjHzpT59k/VSt TDN0YUuWrBNh!


Clever segue to lead into what used to be a separate presentation: Troubleshooting

•  Like I said, turning on validation is easy. The hard part is troubleshooting other people’s DNSSEC mistakes when they get caught by validating resolvers.

•  Often leads to claims of “it doesn’t work on campus but it does at home--something must be wrong with the campus DNS servers!”

•  Comcast now does validation by default. •  Thank you Comcast for reducing instances of the above complaint! •  But it does mean that DNSSEC is for real. If you mess up your

DNSSEC, ~1.7 million people won’t be able to resolve your zone!

•  You will need more sophisticated tools than nslookup. dig will be sufficient, but DNSviz and other tools will help as well.


Let’s get started

•  Let’s look at dig and how it can solve some basic DNS problems.

•  Why dig? •  de facto first-tier DNS troubleshooting tool •  shows output from servers with relevant flags and other important

information •  part of BIND distribution, and widely available •  people on mailing lists (bind-users@ dns-operations@) will both

use and understand dig output.


Let’s get started

•  Why not nslookup? •  It’s old. Really old. •  It masks important diagnostic information. •  Frustratingly, a lot of users will initially report a problem with

nslookup. You’ll quickly see how useless this is. •  It used to use deprecated DNS protocol extensions to work.

(Fortunately that has been excised.) •  In some circumstances, it gives out just plain wrong information.


nslookup problems

•  Why not nslookup? •  Let’s try querying an authoritative-only DNS server with nslookup:

michael@michaels-macbook-pro:~$ nslookup

> server adns1.berkeley.edu

Default server: adns1.berkeley.edu

Address: 2607:f140:ffff:fffe::3#53

Default server: adns1.berkeley.edu

Address: 128.32.136.3#53

> cisco.com

Server: adns1.berkeley.edu

Address: 2607:f140:ffff:fffe::3#53

** server can't find cisco.com: NXDOMAIN


nslookup problems

•  Wow, it knows IPv6. •  But it gets the answer wrong! •  NXDOMAIN means that the domain cisco.com doesn’t exist at all! •  (I am sure that’s news to them.) •  Let’s see what dig says instead:


nslookup problems

michael@michaels-macbook-pro:~$ dig cisco.com @adns1.berkeley.edu

; <<>> DiG 9.4.3-P3 <<>> cisco.com @adns1.berkeley.edu

;; global options: printcmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 53060

;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; WARNING: recursion requested but not available

;; QUESTION SECTION:

;cisco.com. IN A

;; Query time: 67 msec

;; SERVER: 2607:f140:ffff:fffe::3#53(2607:f140:ffff:fffe::3)

;; WHEN: Tue Mar 8 21:19:48 2011

;; MSG SIZE rcvd: 27


dig benefits




;; Got answer:





;cisco.com. IN A



;; WHEN: Tue Mar 8 21:19:48 2011



dig benefits




;; Got answer:





;cisco.com. IN A



;; WHEN: Tue Mar 8 21:19:48 2011



Understanding dig output: flags

•  What are these flags?

•  Defined originally in RFC 1035: 1 1 1 1 1 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ | ID | +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ |QR| Opcode |AA|TC|RD|RA| Z | RCODE | +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ | QDCOUNT | +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ | ANCOUNT | +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ | NSCOUNT | +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ | ARCOUNT | +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+


Understanding dig output: flags

•  What are these flags? Initial RFC 1035 definitions: •  qr: Query Response (response to query) •  aa: Authoritative Answer •  tc: TrunCated (answer is truncated and you need to fall over to TCP)

•  rd: Recursion Desired: In a query, this specifies that the querier wants the server to perform recursion. Its value is copied into the response from the server.

•  ra: Recursion Available: The server is willing to do recursion for this client. Of course, not all servers perform recursion, and those that do, often do not do so for every client. It is generally a best practice to restrict recursion to known clients.

•  The presence of the RD and the absence of the RA flag prompts dig to issue a more visible warning: “recursion requested but not available.


Understanding dig output: status

•  The “status” portion of the message is derived from the RCODE. Some common RCODEs are:

•  NOERROR: (0) - no error condition. Does not imply that there is an answer to the query.

•  FORMERR: (1) - format error. The name server did not understand the query.

•  SRVFAIL: (2) - server failure. The server couldn’t process the query due to a variety of reasons, which will be discussed in a later slide.

•  NXDOMAIN: (3) - “name error” according to RFC 1035. The domain in which the query was made does not exist. Not to be confused with an empty NOERROR response!

•  NOTIMPL: (4) - not implemented. The name server basically understands the query, but doesn’t implement the query type or some other aspect of the query.

•  REFUSED: (5) - refused. The server refuses to answer the query, for a variety of reasons.

•  These are the RCODEs defined by RFC 1035. There are others having to do with dynamic updates and other additions to the DNS protocol. We may discuss some of these later.

•  RCODEs can give significant clues as to why a name server isn’t behaving the way you want it to, and dig is one of the few tools that consistently and accurately reports the RCODE.


A simple dig exercise

•  Let’s figure out if a name server is answering authoritatively for a zone. •  There are a few ways to do this, and it gives us a chance to play with some

of the many dig options. dig options that affect the query that dig makes or the output that it displays are usually prepended with a “+” instead of the usual UNIX “-”. There are some dig options that use the “-” and these generally affect the behavior of the program itself.

•  Let’s query some name servers and try to interpret the dig output to see if they are authoritative for a zone. We’ll also get a good feel for actual dig output of DNS records.



•  I am going to open a terminal window now and do some dig queries from the command line.

•  Whoops, that’s a lot of output. (Wait until we try ‘dig any berkeley.edu!)

•  There are ways to make it shorter.



•  Wow, that really is short.

•  It’s useful for scripting and piping because it doesn’t include any extraneous information or boilerplate, like, say nslookup does. But it also doesn’t include all of the useful information that can make dig helpful for troubleshooting.

•  Instead, we have can use other flags to have dig omit the additional and/or authority sections.

•  Okay, let’s look at the +noauthority output.


A simple dig exercise ; <<>> DiG 9.4.3-P3 <<>> +noadditional berkeley.edu ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35444 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 10 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;berkeley.edu. IN A ;; ANSWER SECTION: berkeley.edu. 600 IN A 169.229.131.81 ;; AUTHORITY SECTION: berkeley.edu. 172800 IN NS adns2.berkeley.edu. berkeley.edu. 172800 IN NS ns.v6.berkeley.edu. berkeley.edu. 172800 IN NS aodns1.berkeley.edu. berkeley.edu. 172800 IN NS phloem.uoregon.edu. berkeley.edu. 172800 IN NS sns-pb.isc.org. berkeley.edu. 172800 IN NS adns1.berkeley.edu. ;; Query time: 16 msec ;; SERVER: 128.32.206.9#53(128.32.206.9) ;; WHEN: Sat Mar 12 18:31:28 2011 ;; MSG SIZE rcvd: 404


A simple dig exercise ; <<>> DiG 9.4.3-P3 <<>> +noadditional berkeley.edu ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35444 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 10 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;berkeley.edu. IN A ;; ANSWER SECTION: berkeley.edu. 600 IN A 169.229.131.81 ;; AUTHORITY SECTION: berkeley.edu. 172800 IN NS adns2.berkeley.edu. berkeley.edu. 172800 IN NS ns.v6.berkeley.edu. berkeley.edu. 172800 IN NS aodns1.berkeley.edu. berkeley.edu. 172800 IN NS phloem.uoregon.edu. berkeley.edu. 172800 IN NS sns-pb.isc.org. berkeley.edu. 172800 IN NS adns1.berkeley.edu. ;; Query time: 16 msec ;; SERVER: 128.32.206.9#53(128.32.206.9) ;; WHEN: Sat Mar 12 18:31:28 2011 ;; MSG SIZE rcvd: 404



•  Note that the aa bit is set in this output. •  Note also that if we query the same server over and over again, the

TTLs do not decrement. •  [Demonstrate the use of multiple queries to a recursive server to

show how ttl decrements and aa bit not set.]

•  We could also use +norecurse flag. •  [demonstrate its use]

•  Using dig, we can see if a server is answering authoritatively for a domain.

•  As you may know, when a server is not answering authoritatively, but it is supposed to be, it is lame.



•  Setting query types: •  Note that omitting a query type leads to the default of type A. (No

AAAA by default.) •  To query for other types, simply include them on the command line. -  dig ns berkeley.edu -  dig mx berkeley.edu -  dig aaaa berkeley.edu -  dig txt berkeley.edu

•  Order does not matter, for the most part: -  dig ns berkeley.edu @adns1.berkeley.edu == -  dig @adns1.berkeley.edu berkeley.edu ns

•  Try this: dig any berkeley.edu on different servers...


A troubleshooting scenario

•  A user reports that they cannot resolve a domain (let’s say borkeddns.com) from your name servers, but they can resolve it from their home or ISP name server.

•  If you’re on a campus or you are a big ISP, you’ll probably suspect caching issues.

•  But what’s causing them? And more importantly, whose fault is it? •  Of course, if you’re validating DNSSEC (and others are not), it may

be a validation failure. We’ll talk about that later, but what if that’s not the case?

•  dig is going to be your best friend.


A troubleshooting scenario

•  First, let’s figure out what the authoritative servers say about the zone. You can try this on your laptops if you have dig installed.

•  I can never remember the .com name servers, so let’s start out querying the root:

•  dig ns borkeddns.com @f.root-servers.net •  Oh, yeah, here’s one: a0.org.afilias-nst.info. (Since the root servers are not

authoritative for .org, we received a downward referral.)

•  dig ns borkeddns.com @a.gtld-servers.net •  dig a borkeddns.com @burnttofu.net •  Hmmm. Note the authority section (if there is one). Else:

•  dig ns borkeddns.com @burnttofu.net


Scenario: authority mismatch

•  Oooh, the NS records are different in the parent zone (org) than in the child zone!

•  Moreover, the NS records in the child zone don’t work! Try:

•  dig a borkeddns.com @servfail.borkeddns.com •  dig a borkeddns.com @timeout.borkeddns.com •  (In real life, the NS names probably won’t be such a giveaway, which is why

you need dig to tell you what’s going on!)


Sidebar

•  What does SERVFAIL really mean? •  I am not a recursive server and I don’t know nuthin about the zone in

which ur querying. This may also result in a REFUSED or in an upward referral (deprecated).

•  It appears that my administrator is trying to configure the zone, but it is broken, missing, or horribly misconfigured and I just can’t give you an answer.

•  It’s configured in me correctly, but I am a slave to an upstream master, and I haven’t (ever?) been able to do a zone transfer from the master, and/or the zone has expired.

•  In a DNSSEC context: I am configured to do validation and the records I am trying to validate are signed, but they fail validation.

•  Other possibilities? [ask dns-operations]



•  What’s happening here? •  First, note that the consensus interpretation of the DNS standards is that

the parent (com) is authoritative for the existence of the NS records, but the child (borkeddns.com) is authoritative for the contents of the NS records.

•  So, most caching name servers will be child-centric. They will replace the contents of the NS records that they got from the parent with those they got from the child authoritative server.

•  But since the replacement NSes don’t work, the zone has effectively been poisoned--not in a malicious way, but simply in an incompetent way.

•  This has a further implication for the resolution of the A record for borkeddns.com. Look at the output again.

•  [Next slide will show output of dig showing lower ttl for A record than NS records.]



•  Here’s the problem: The TTL for borkeddns.com’s A record is only 600 seconds, but the TTL for the (bad) NS records is 2 days. So, the A record expires out, but the bad NS records are still in the cache. The caching server tries to look up the A record again but can’t because it has bad data.

•  This is not a rare problem. It often happens with the admin for a domain changes his/her name servers. The admin updates the registry records but forgets to fix the NS records in the zone itself. When the “old” servers are shut down or repurposed, the problem arises.

•  Authority mismatches in general are bad, and this is an example of the most egregious error.



•  Okay, so why did the original report say that my name servers were broken and others were working?

•  My servers may be busier and have larger caches. They may have already cached the information whereas a home CPE router may not.

•  Some implementations of caching servers are “parent-centric” and will stick with the parent’s NS record contents. This is arguably a standards violation.

•  In the past some servers, upon finding lame delegations, would delete the NS record that was lame and, if all authoritative NS records were deleted, they would refetch the NS records from the parent. This can create big problems with looping queries if there are no NS records in the parent that work! But in this case, they would appear to the client to be working correctly.

•  At any rate, the problem is with the remote domain.

•  Note that dig gave us all of the information we needed to solve this problem. Could nslookup have done that?


A couple short-cuts

•  Now that we went through all of those dig commands to look for the authoritative name servers for borkeddns.com, let me show you a short-cut:

•  dig +trace ns borkeddns.com •  Iterates from the root downward and shows all of the authority records.

Here you can easily see the authority mismatch.

•  In some cases the problem is not an authority mismatch, but a case where one authoritative server isn’t properly getting updated versions of the zone file. In this case, the serial number on the SOA record of the zone would be different on that server. There’s an easy way to check this with dig:

•  dig +nssearch borkeddns.com •  It’s sometimes fun to try this on TLDs or ccTLDs as they are in the

process of updating. •  It’s also useful to check zones you manage periodically to ensure they are

getting updated zones as you expect.


A different scenario [optional]

•  A departmental IT administrator reports that some of her users are reporting intermittent problems resolving the vendor management server of a commercial software package her group is using. Occasionally it works, but frequently, the application hangs and reports a “DNS time-out” error. She therefore assumes that it is a DNS problem.

•  Of course, you have no idea what the application is trying to do, so where do you start?

•  Sniff the wire!


A different scenario

•  You probably know about the major tools used to sniff an ethernet link, especially tcpdump and wireshark.

•  tcpdump -v can give useful DNS information. •  So can wireshark or tshark (command-line version of wireshark).

•  You may not know about dnscap and nmsg. •  dnscap: a pcap-based tool that is exclusively for DNS traffic. It can do

regular-expression matches on the DNS message itself. •  nmsg: an extensible tool that can accept various plugins for different types

of traffic (including DNS). One of the plugins can produce an output format similar to dig.


Output comparison for wire DNS traffic

[show output for tcpdump and tcpdump -v]



•  [show tshark output here]



•  [show dnscap -x -g output here (may also need dnscap -x -6 -g)]



•  [show nmsgtool -i <int>+ -V ISC -T dnsqr output here]


Back to the scenario

•  Note what we saw from the various tools: The application was making queries of type ANY. That can easily cause responses of length greater than 512 bytes.

•  Originally, RFC 1035 specified that UDP DNS messages must be restricted to 512 bytes. If the message exceeded 512 bytes, implementations must fall over to TCP. Of course, sometimes people block TCP/53 because they think it is only necessary for zone transfers.

•  RFC 2671 specified an extension, EDNS0, which allows for much larger responses over UDP.

•  Can lead to UDP fragmentation. But some people block UDP fragments!

•  [demonstrate dig +bufsize=4096] Note that some versions of dig may do this by default.

•  Note also dig output in a TC situation. [demonstrate this.]


Looking at DNSSEC output

•  We’ll get back to the scenario again in a bit. Let’s first look at DNSSEC output with dig. Here are the DNSKEYs for berkeley.edu as reported by dig:

berkeley.edu. 300 IN DNSKEY 257 3 10 AwEAAbxkCCvANJ8oHUnvYuugq1lbQagoKZb40B1c8mY9eZ3BpGxcoACy mCkiIWVnR7ebz8bqfrUXyaFgSL391rX2eQSq75h50llokLnVbu9c/uWH /9dQ+5Wofc8O68Vvifdxq841/IkdEc9mKlKSDtgFyVfQWkv/GeZbupK9 LMuuv71kCtz+P+3tEDwdsZ8Ay00RHG4R0y9K8G+Bac8sMAtL9+3ZHpTl ySKj8OZOIW11/TADtomdZ2vdp/WruG5gn1t7omD83QQurmxjniN0aTs8 imsuvXOst8sYZYMaBZSQeK5kcEltVFcHdRzc1yoA57f8Sjpek7DgvA3U 0ctXJVOo1Jk=

berkeley.edu. 300 IN DNSKEY 256 3 10 BQEAAAAB61Gwv6rGmQ2hNNiM+eVzJe3IfFYemDanzVHpjz4cvBUFk8SS ai6fI9MYRh2QbFQHAAQLPSY7ZnGkG+y+nXue4S808f4zk+DX/vujKCap BzktbWnyeRP/VHV5N0MZXPD/qvstSS2lmOuRMQYMpTohIzJKvi3dyeZQ sBqJR9PNpZc=



•  sometimes, it’s easier to look at key material in the multiline format. We can use the ‘+multi’ option to dig to get this format:

$ dig +multi +noall +answer dnskey berkeley.edu berkeley.edu. 300 IN DNSKEY 256 3 10 (

BQEAAAAB61Gwv6rGmQ2hNNiM+eVzJe3IfFYemDanzVHp jz4cvBUFk8SSai6fI9MYRh2QbFQHAAQLPSY7ZnGkG+y+ nXue4S808f4zk+DX/vujKCapBzktbWnyeRP/VHV5N0MZ XPD/qvstSS2lmOuRMQYMpTohIzJKvi3dyeZQsBqJR9PN pZc= ) ; key id = 42697

berkeley.edu. 300 IN DNSKEY 257 3 10 ( AwEAAbxkCCvANJ8oHUnvYuugq1lbQagoKZb40B1c8mY9 eZ3BpGxcoACymCkiIWVnR7ebz8bqfrUXyaFgSL391rX2 eQSq75h50llokLnVbu9c/uWH/9dQ+5Wofc8O68Vvifdx q841/IkdEc9mKlKSDtgFyVfQWkv/GeZbupK9LMuuv71k Ctz+P+3tEDwdsZ8Ay00RHG4R0y9K8G+Bac8sMAtL9+3Z HpTlySKj8OZOIW11/TADtomdZ2vdp/WruG5gn1t7omD8 3QQurmxjniN0aTs8imsuvXOst8sYZYMaBZSQeK5kcElt VFcHdRzc1yoA57f8Sjpek7DgvA3U0ctXJVOo1Jk= ) ; key id = 12834



•  Notice how the multiline format gives us the key id of the two keys. That’s very important when we start looking at signatures.

•  Let’s look at the other fileds: •  256/257 -  256: The SEP bit is not set on the key. This is typically used as a zone-

signing key (ZSK). -  257: The SEP bit is set on the key. This is typically used as a key-

signing key (KSK). •  3: Protocol. 3 == DNSSEC. For DNSSEC DNSKEYs, this must always be

set to 3.

•  10: Algorithm. In this case the algorithm is RSASHA512. (See http://www.iana.org/assignments/dns-sec-alg-numbers/dns-sec-alg-numbers.xml)



•  Look at RRSIGs:

adns1.berkeley.edu. 3600 IN RRSIG AAAA 10 3 3600 20110415171839 20110311171839 42697 berkeley.edu. BmSJ87lF83V77yrbSrBzs4MZMxEaDd0kEcGbDHxDhvVzESn+JwyT8o3d gV9XQdgta9vQfAgTWVb7PUAC8NO+Gi0T1aYZ5QcEBIff0GmHIA8mp5hU HuFl3C36wrPboSiW5+7CAMPKqgjfqU06UDr0FCmm+ELyx2SVwcWg1GGo Rys=

adns1.berkeley.edu. 172800 IN RRSIG A 10 3 172800 20110415171839 20110311171839 42697 berkeley.edu. K9HyyI+K1DxWd6oY3u+Lh6/p5cliPTvVfkv8VQvHQ5hR7417CuHtLNfO FfCZx238IwQyiWbFPAPgEBTeUIwEL8oGNcj4Dx3le4LMkf61MeTJSY0V tQBPYLwdiSRFIwu3HPZkxERGGHZ3ih/sbVkN/RkPcI3F957u0AoaOIdk H/g=

•  You can see the usual TTL, CLASS, and RRTYPE fields. Next is the RRTYPE of the record that the RRSIG is signing, then the algorithm, protocol (DNSSEC==3 again), and the TTL of the record being signed. Next comes the expiration date of the signature and the date/time that the signature was created, followed by the key id of the key used to sign the record (told you it was important!) and the zone that holds the record.



•  We’ll talk more about signature expirations later--it’s generally going to be a big part of DNSSEC troubleshooting.

•  There’s one more record I want to show you: the NSEC record:

adns1.berkeley.edu. 300 IN NSEC adns2.berkeley.edu. A AAAA RRSIG NSEC

adns1.berkeley.edu. 300 IN RRSIG NSEC 10 3 300 20110415171839 20110311171839 42697 berkeley.edu. asjx3nEpKsabaW+K/g36ybgDrbZysOG0pdAItQgJRRhy5R5saInUsrue WNoEON5fQhDhvzTv+kZXKseX3D7oxOSLVwHI1V1BfLcjUwTNIBmCb2xS E6gdy7V7q68dEc6brmFDbjr2CE9QNS9/LdTSC8eOHnjeJbDvPc4j8apb KH8=

•  Since you already know the DNSSEC basics, you probably understand that the NSEC record provides proof of non-existence. What the record above basically says is “there are A AAAA RRSIG and (of course) NSEC records for adns1.berkeley.edu and nothing else until adns2.berkeley.edu, which is the next RR.”



•  As you can see, DNSSEC adds a bunch of records and creates a lot of additional output in DNS responses. Try playing around with dig some more to see how large the DNS responses are.

•  dig any <domain> will give you all of the records for a domain or a hostname, including DNSSEC-related records.

•  dig +dnssec [type] <name> will give you just the RRSIG for the records you requested. The +dnssec sets the DO (DNSSEC OKAY) bit on the query.

•  Note the sizes of these messages, remembering that messages over 512 bytes require either TCP or EDNS0.

•  As more domains sign their zones, responses are gonna get bigger!



•  You find out that your IT administrator has a firewall. Using nmsg [demo this], we can see that the TC (truncated) bit is set on the responses. That means that the response could not fit inside a 512-byte message, and tcp failover is required.

•  You can also see the same thing with the +ignore flag on dig.

•  Try dig +ignore any berkeley.edu •  You can also see from these responses that there is DNSSEC data in the

response. You surmise that the vendor has started signing their zones and that may have “caused” the problem.

•  Okay, what question do you ask the IT administrator?



I will not block TCP/53 and I will allow UDP fragments I will not block TCP/53 and I will allow UDP fragments


I will not block TCP/53 and I will allow UDP fragments



I will not block TCP/53 and I will allow UDP fragments




•  But remember that the problems were intermittent. Wouldn’t that imply that the firewall was not to blame? After all, the FW should be blocking all of the responses.

•  However, caching name servers will sometimes respond to ANY queries entirely from cache. They may not have all of the records in their cache, so they will respond with smaller messages.

•  Some services (e.g. OpenDNS) filter their ANY responses to keep them small. If some of the users in this department configured OpenDNS as one of their resolvers, you might see occasional success.

•  But there is another explanation.


Scenario: Large responses

•  You should also ensure that all of the authoritative name servers are able to send you the large response. Test each one of them using dig. If a majority of the authoritative servers for a domain are themselves blocking TCP (or some external firewall is blocking TCP), then you will get a similar problem.

•  In general, applications should not do queries for type ANY, but some do. Unfortunately, it’s not your job to argue with their developers about what they should and should not do. We do see applications in the wild that attempt to perform these queries.

•  When you sign your zones, you can generally expect to have problems with older versions of qmail attempting to send mail to your users.

•  You don’t need DNSSEC to get large responses. As of March 2011, ‘dig any paypal.com’ and ‘dig any microsoft.com’ yielded responses over 512 bytes.


DNSSEC overview

•  Large responses were our first foray into DNSSEC in this tutorial, and they will generally be the first problem you run into. Since the root and other zones have been signed, the large response issue (people with out-of-date or incorrect firewall rules) has diminished. But there may be more of them lurking out there.

•  We have seen a bunch of DNSSEC output of dig. Let’s now cover the rest of the bits of DNSSEC we’re likely to encounter when troubleshooting.


DNSSEC overview

•  Flags: •  AD (defined in RFC 2535 “old DNSSEC”): Authenticated Data: The

records have been validated. •  DO: DNSSEC okay: Used in a query to request DNSSEC records

pertaining to the query. •  CD: Checking Disabled: This is set in the query when one doesn’t

want a validating name server to attempt to validate the requested records. The bit is also set in the response to indicate that checking has indeed been disabled. This is very useful for troubleshooting as one might imagine.

•  Note that CD and DO can be combined (and often are in a forwarding setup).


DNSSEC overview

•  Validation terms: •  Valid/Secure: The zone is signed, I have a trust-anchor (or chain of

trust) for it, and the signatures validate. I return the answer with the AD bit set.

•  Insecure: Either the zone is not signed or the chain of trust is broken somewhere. I return the answer but don’t set the AD bit.

•  Bogus/Validation Failure: The zone is signed, I have a trust-anchor (or chain of trust) for it, but the signatures don’t validate. I return a SERVFAIL, unless the CD bit is set on the query, in which case I return the answer and set the CD bit on the answer (and certainly don’t set the AD bit). -  Some implementations can also be configured to ignore

validation failures for certain zones and instead treat them as insecure.

•  I will use secure/insecure/bogus to refer to these three situations.


DNSSEC overview

•  Useful dig flags for DNSSEC: •  +dnssec: Sets the DO bit, which requests DNSSEC records and, if

the server validates, requests that the AD bit be set if the zone/records are secure.

•  +cdflag: Sets the CD bit, which disables validation of a validating name server.

•  Both may be combined. Again, this is useful for troubleshooting.

•  Note also that DNSSEC requires EDNS0 support. dig will use EDNS0 (default 4096 byte buffer) even if you don’t specify +bufsize=4096.

•  Dig will include the following, per RFC 2671: ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 4096


DNSSEC signature scenario

•  Let’s use dig to troubleshoot a fairly common (unfortunately) DNSSEC problem.

•  dig +dnssec www.expired.brokendns.net •  [validating server for folks to use will be provided by me]

•  What happened? SERVFAIL. Unfortunately that could be related to a few things. To rule out (or in) DNSSEC, what query would you use?

•  What does the output tell you?


DNSSEC signature scenario

•  The signatures have expired, as you can see from their timestamps. •  This is a common problem, especially where signing is done

manually. •  There are also web-based tools that can detect this issue.

•  dnsviz.net •  dnssec-debugger.verisignlabs.com •  Both of these are excellent tools for examining DNSSEC problems.

Let’s run both on our expired.tinkerdork.org domain. •  [spend time demoing the two tools.]


DNSSEC delegation scenario

•  dnsviz and dnssec-debugger are great tools, but they can’t always find every problem.

•  Here’s an example of one where they can’t find a problem (or where they couldn’t).

•  Try dig +dnssec id.id.brokendns.net

•  SERVFAIL, but we don’t know why. •  Now try dnssec-debugger first. It looks like it’s just an unsigned

child zone of brokendns.net. •  Let’s try dig next: •  dig ns id.brokendns.net @tinkerdork.org

•  dig id.id.brokendns.net @tinkerdork.org

•  Things look okay.



•  Now let’s look for DS records. Maybe the zone is supposed to be signed but isn’t:

•  dig ds id.brokendns.net •  Okay, there are no DS records, so it should be a regular unsigned zone.

But look closely at the “status” field in dig, which gives the RCODE. It says NXDOMAIN. Recall from an earlier slide what NXDOMAIN means--it means that the entire domain of id.brokendns.net doesn’t exist!

•  Repeat the above dig query with the +dnssec flag. Note the NSEC record. id.brokendns.net doesn’t exist and DNSSEC proves it!

•  So why does the same DNS server give me a response when I query it without DNSSEC? Why do non-validating servers like google properly resolve the domain? (Should we cue the Twilight Zone music?)

•  Okay, let’s now see what dnsviz.net says for this domain.



•  Hmm, dnsviz says that there are no delegation records. But the tinkerdork.org nameserver says that there are NS records. (Initially, dnsviz was not able to detect such errors.)

•  Remember that delegation records must exist in both the parent and child--and they should be the same, as we have seen.

•  Because tinkerdork.org is authoritative for the parent (brokendns.net) and the child (id.brokendns.net), the delegation issue generally gets swept under the carpet...until DNSSEC comes along.

•  Although this is a common mistake, people have been getting away with it for so long, they’re not likely to understand that they have been misconfiguring their DNS all along!

•  Even without DNSSEC, their party will be over if or when they have another party provide secondary service for parent and not the child. Then the id.brokendns.net will intermittently not exist, regardless of the DNSSEC status. DNSSEC does us a favor by forcing the issue.


DNSSEC NSEC scenario

•  You are experiencing intermittent problems resolving another unsigned child domain of a signed parent. Sometimes it resolves and sometimes you get SERVFAILs.

•  Such intermittent problems often indicate that some (but maybe not all) authoritative servers are giving out bad information. This is usually a good task for dnsviz, since it checks to make sure that all of the authoritative name servers are responding properly. Let’s try it. [demonstrate dnsviz on a domain]

•  dnsviz shows us that there a problem with some of the NSEC records in the parent, but it doesn’t tell us exactly what. We need dig for that.

•  Let’s use dig to find the NSEC record that each server is giving out. We may need to set +cdflag to ensure that we can see the records, in case they’re failing validation [demonstrate].


DNSSEC NSEC scenario

•  Note that the NSEC records have different case (capitalization) on some [one?] of the authoritative servers. Because the case is different, but the signatures are the same, the NSEC records fail validation (bogus) when retrieved from some of the servers.

•  Because the NSEC record cannot be validated, it cannot prove the non-existence of the DS record for the child zone, and a validating server will return a SERVFAIL for the child zone.

•  This is sometimes caused by a bug in the way that implementations handle case in zones and the shorting (de-FQDN-ing?) of domain names. This bug has been fixed in known implementations.

•  In this case, dnsviz pointed us in the direction to look, but we had to use dig to get the full answer.


Optional

•  [If time and interest, demonstrate how I used dnscap to discover caching bug.]


Wrapping it up

•  No single too does everything, but if you want to troubleshoot difficult DNS, and especially DNSSEC, problems, you need to move beyond nslookup!

•  dig should be your go-to tool, even for everyday use, so that you get a good feel for its options and syntax.

•  DNS troubleshooting has always been hard, but it is not insurmountable, even with DNSSEC.

•  In fact, DNSSEC forces us to avoid bad behavior that we would otherwise get away with.

•  Troubleshooting DNS and DNSSEC will be an even more important skill in the coming months and years.

•  Make sure you have the right tools!


Appendix

•  dig flags: The following are the flags to dig that we discussed in the tutorial and what they do.

•  [To be written; I will include all of the flags in the tutorial and brief descriptions of what they do.]

•  http://dnsviz.net

•  http://dnssec-debugger.verisignlabs.com •  ‘man dig’ from your favorite command line

Documents

DNSSEC: Signing, Validating, and Troubleshooting · Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science DNSSEC: Signing, Validating, and Troubleshooting