NBDE...NBDE: How I could have slept better at night Chuck Mattern Principal Solution Architect Red...

NBDE:How I could have slept better at night

Chuck MatternPrincipal Solution ArchitectRed Hat

Red Hat Customer 18 years

Linux User and Admin (TAMU, Slackware, Red Hat (& Enterprise), SuSE, Yggdrasil, Mandrake, Debian, CentOS, Scientific, Fedora)

26 years

Unix User and Admin (Coherent, UNIXWare, DG-UX, HP-UX, AT&T B3, Solaris, AIX, OpenBSD, Dynix/PTX, DEC Unix, Ultix, SCO, PrimeOS)

27 years

VMWare ESX (Engineer & Architect) 5 years

Indus International (Unix Admin, Certified Solaris Admin (OS, Networking and Storage) 1998)

1.5 years

Home Depot (Loss Prevention Supervisor, Programmer, Sys Admin, Architect, Principal Engineer, Red Hat Certified Engineer (RHEL 4 (2005), 6 (2011))

27 years

The Paradies Shops (Sr. Manager: Server, Network, Telephony, Desktop ) 1.5 years

Red Hat (Solution Architect, Red Hat Certified Engineer (RHEL 7 (2016)) ~5 years

My old intro

I’m Irish, Italian and Sysadmin-ish● We tend to talk with our hands● We get excited● We are passionate● We like to share stories

Preface: Some core concepts

“Only in self discipline will you ever find freedom”-Hon. James A. Walsh et al

“Nobody is so horrible that he can’t be the perfect bad example.”-John Kelly

“There but for the grace of God go I.”-Hon. James A. Walsh et al

My Cautionary Tale

➢ 2,000+ sites across the US (including Puerto Rico, Hawaii and Guam)➢ 4,000+ ESX hosts➢ 2,000+ iSCSI storage units➢ 2,000+ Windows 2003 VMs➢ 10,000+ RHEL VMs➢ Global deduplicating compressing backup/recovery solution living on

the same storage unit as the other VMs and replicating to a central site

➢ Fractional T1 to each location sharing credit auth and VOIP➢ No local technical staff➢ What could possibly go wrong?➢ ...oh yeah, my support team was 5 Engineers...

image via Peakpxhttp://www.peakpx.com/571722/man-in-black-wet-suit-on-sea

Enter the PRS

Portable Recovery Server➢ Run! Don’t walk…

➢ Grab the best castoff desktop you can find in the basement➢ Snag two 1TB SAS disks a spare NIC and a gig of RAM from Microcenter➢ Base install of RHEL4, mirroring the disks

➢ Encrypt the root volume with luks and use something tough like K&tx#vQ2*HW@9ucB!➢ Remember, it’s a $50-$100M a year business, in a box!

➢ Expose all spare disk via NFS➢ Mount that up to your ESX host via primary NIC➢ Build out a temporary recovery VM via ESX on the NFS share➢ Replicate backup data➢ Munge through and rename, re-IP everything under the covers➢ Slap the remote location IP on the secondary NIC ‘cause DHCP lived on one of the dead VMs (can

you say down hard?)➢ Shutdown and pack it in a box you found in the basement with styro-peanuts you stole^H^H^H

borrowed from the shipping folks➢ Drive like a maniac to Delta Dash then…➢ ...wait...

What is LUKS?

➢ Linux Unified Key Setup ➢ from Clemens Fruhwirth in 2004➢ Originally for Linux, now there are

➢ Android (yeah I know it’s Linux under there)➢ Windows➢ maybe elsewhere?

https://en.wikipedia.org/wiki/Linux_Unified_Key_Setuphttps://gitlab.com/cryptsetup/cryptestup/

Translating: It’ll be OK, I promise

By Dallastechline, Inc. [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

By Servershop24 [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)],from Wikimedia Commons

➢ With a DR solution based on a scavenged desktop I had difficulty establishing credibility with my end customer even though I had a well thought out technical solution to the issue at hand.

➢ Talking a non-technical user through decrypting the root volume with a password such as K&tx#vQ2*HW@9ucB! Did not make things any easier

image via Peakpxhttp://www.peakpx.com/571722/man-in-black-wet-suit-on-sea

What is NBDE?

Network Bound Disk Encryption➢ Linux systems can decrypt volumes, even root volumes, over the

network➢ Based on clevis and tang

➢ clevis framework for the client side➢ inserts into dracut➢ has several “pins”➢ https://github.com/latchset/clevis

➢ tang for the server side➢ one of the clevis “pins”➢ https://github.com/latchset/tang

License: CC0 Public Domain Robust Clevis On Vehicle

Where can I use NBDE?

● Laptops (duh…)● Workstations● Servers

○ yep, even portable ones….

Logical View of Clevis and Tang

Architectural View

Server Installation

[root@tang3 ~]# yum install -y tang[omitted]Installed: tang.x86_64 0:6-1.el7

Dependency Installed: http-parser.x86_64 0:2.7.1-5.el7_4 jose.x86_64 0:10-1.el7 libjose.x86_64 0:10-1.el7

Complete![root@tang3 ~]# systemctl enable tangd.socket --nowCreated symlink from /etc/systemd/system/multi-user.target.wants/tangd.socket to /usr/lib/systemd/system/tangd.socket.[root@tang3 ~]# systemctl status tangd.socket

● tangd.socket - Tang Server socket Loaded: loaded (/usr/lib/systemd/system/tangd.socket; enabled; vendor preset: disabled) Active: active (listening) since Tue 2018-10-16 06:01:23 UTC; 11s ago Listen: [::]:80 (Stream) Accepted: 0; Connected: 0

Oct 16 06:01:23 tang3.mobile.roninprinciples.com systemd[1]: Listening on Tan...Oct 16 06:01:23 tang3.mobile.roninprinciples.com systemd[1]: Starting Tang Se...Hint: Some lines were ellipsized, use -l to show in full.[root@tang3 ~]# firewall-cmd --add-service=httpsuccess[root@tang3 ~]# firewall-cmd --add-service=http --permanentsuccess[root@tang3 ~]#

Server Installation and Configuration

Client Installation

[root@clevis ~]# yum install -y clevis-dracut[omitted]Installed: clevis-dracut.x86_64 0:7-8.el7

Dependency Installed: clevis.x86_64 0:7-8.el7 clevis-luks.x86_64 0:7-8.el7 clevis-systemd.x86_64 0:7-8.el7 jose.x86_64 0:10-1.el7 libjose.x86_64 0:10-1.el7 libluksmeta.x86_64 0:8-1.el7 libpcap.x86_64 14:1.5.3-11.el7 luksmeta.x86_64 0:8-1.el7 nmap-ncat.x86_64 2:6.40-16.el7 tpm2-abrmd.x86_64 0:1.1.0-9.el7 tpm2-tools.x86_64 0:3.0.4-1.el7 tpm2-tss.x86_64 0:1.4.0-1.el7 tpm2-tss-devel.x86_64 0:1.4.0-1.el7

Complete![root@clevis ~]#

Client Installation: Software

[root@clevis ~]# cryptsetup luksDump /dev/vda2LUKS header information for /dev/vda2

Version: 1Cipher name: aesCipher mode: xts-plain64Hash spec: sha256Payload offset: 4096MK bits: 512MK digest: 58 e6 af 4c 89 a8 05 f1 f9 fc 8d 11 52 d8 44 60 c0 1c d7 43 MK salt: d8 c2 51 ae cd e7 3b d5 f7 9b 11 24 dd 20 b9 3f 10 49 43 5e 11 79 16 f0 c1 35 6a 62 27 0e b3 96 MK iterations: 13000UUID: 80e99979-147b-45fd-88cd-7e8ec6b195c2

Key Slot 0: ENABLEDIterations: 98308Salt: a6 6a 9f 45 a0 fb 11 f2 a4 e0 a8 02 58 25 a7 b6 0a 54 04 51 c8 5a ce 5f 5a 7f c4 0e 87 e4 fc 68 Key material offset: 8AF stripes: 4000

Key Slot 1: DISABLEDKey Slot 2: DISABLEDKey Slot 3: DISABLEDKey Slot 4: DISABLEDKey Slot 5: DISABLEDKey Slot 6: DISABLEDKey Slot 7: DISABLED[root@clevis ~]#

Client Installation: luks Status

[root@clevis ~]# clevis luks bind -d /dev/vda2 sss '{

"t": 2, "pins":

{"tang": [{"url": "http://tang1.mobile.roninprinciples.com"}, {"url": "http://tang2.mobile.roninprinciples.com"}, {"url": "http://tang3.mobile.roninprinciples.com"}

]}

}'The advertisement contains the following signing keys:

TepHUGV79tG8Cs0L9XPQh2s0f8A

Do you wish to trust these keys? [ynYN] yThe advertisement contains the following signing keys:

_tE0s8Q9oMn7gF4Hqhehl9irSac

Do you wish to trust these keys? [ynYN] yThe advertisement contains the following signing keys:

LdsB17ihj8MhRCaM8OiHEKkw2q8

Do you wish to trust these keys? [ynYN] yEnter existing LUKS password: [root@clevis ~]#

Client Installation: Configure clevis

Note: This example assumes a single block devise supporting an LVM volume group. Configurations with multiple block devices will require additional configuration.

[root@clevis ~]# cryptsetup luksDump /dev/vda2LUKS header information for /dev/vda2

Version: 1Cipher name: aesCipher mode: xts-plain64Hash spec: sha256Payload offset: 4096MK bits: 512MK digest: 58 e6 af 4c 89 a8 05 f1 f9 fc 8d 11 52 d8 44 60 c0 1c d7 43 MK salt: d8 c2 51 ae cd e7 3b d5 f7 9b 11 24 dd 20 b9 3f 10 49 43 5e 11 79 16 f0 c1 35 6a 62 27 0e b3 96 MK iterations: 13000UUID: 80e99979-147b-45fd-88cd-7e8ec6b195c2

Key Slot 0: ENABLEDIterations: 98308Salt: a6 6a 9f 45 a0 fb 11 f2 a4 e0 a8 02 58 25 a7 b6 0a 54 04 51 c8 5a ce 5f 5a 7f c4 0e 87 e4 fc 68 Key material offset: 8AF stripes: 4000

Key Slot 1: ENABLEDIterations: 176884Salt: 12 8b 7e cd d8 79 b3 44 19 fd 4c bd 82 84 5d 1f ec aa 60 72 1a 14 8b 65 b1 e1 95 a2 de 3c cc eb Key material offset: 1016AF stripes: 4000

Key Slot 2: DISABLEDKey Slot 3: DISABLEDKey Slot 4: DISABLEDKey Slot 5: DISABLEDKey Slot 6: DISABLEDKey Slot 7: DISABLED[root@clevis ~]#

Client Installation: luks Status

[[root@clevis ~]# luksmeta show -d /dev/vda20 active empty1 active cb6e8904-81ff-40da-a84a-07ab9ab5715e2 inactive empty3 inactive empty4 inactive empty5 inactive empty6 inactive empty7 inactive empty[root@clevis ~]#

Client Installation: luksmeta status

Delivering: It’ll be OK, I promise

By Dallastechline, Inc. [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

By Servershop24 [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)], from Wikimedia Commons

License: CC0 Public Domain Jeff Rowleyhttps://www.flickr.com/photos/jeffrowley/6675136983/in/photostream/

A few of the finer points

➢ No encryption needed in flight➢ Luks key is never transmitted➢ Only the encrypting key is transferred over the wire➢ Encrypted paraphrase is stored in luks header

Encryption Walk Through

[root@clevis ~]# echo 'Good Morning Columbus, Ohio!' | clevis encrypt sss '{ "t": 2, "pins": {"tang": [ {"url": "http://tang1.mobile.roninprinciples.com"}, {"url": "http://tang2.mobile.roninprinciples.com"}, {"url": "http://tang3.mobile.roninprinciples.com"} ] } }' >gmco.jweThe advertisement contains the following signing keys:

TepHUGV79tG8Cs0L9XPQh2s0f8A

Do you wish to trust these keys? [ynYN] yThe advertisement contains the following signing keys:

_tE0s8Q9oMn7gF4Hqhehl9irSac

Do you wish to trust these keys? [ynYN] yThe advertisement contains the following signing keys:

LdsB17ihj8MhRCaM8OiHEKkw2q8

Do you wish to trust these keys? [ynYN] y[root@clevis ~]#

Encrypting a sample passphrase

With two servers down the threshold of 2 out of 3 tang servers cannot be met:

[root@clevis ~]# clevis decrypt <gmco.jweError communicating with the server!Error communicating with the server![root@clevis ~]#

Once at least 2 of the 3 servers are online we can decrypt the passphrase:

[root@clevis ~]# clevis decrypt <gmco.jweGood Morning Columbus, Ohio![root@clevis ~]#

Decrypting a sample passphrase

It’s not just tang for breakfast anymore

Shamir’s Secret Sharing➢ from Adi Shamir➢ Allows for combinations of multiple kinds of pins

➢ tang➢ tpm2➢ http

➢ math too painful for mere mortals➢ think of it as the intersection of RAID and cryptography for now➢ see the Wikipedia link below if you’re a cryptographer,

mathematician or just like pain

https://en.wikipedia.org/wiki/Shamir's_Secret_Sharing

Magical things you can do with SSS

Magical things you can do with SSS

Magical things you can do with SSS

Magical things you can do with SSS

Magical things you can do with SSS

Magical things you can do with SSS

Magical things you can do with SSS

Magical things you can do with SSS

Magical things you can do with SSS

clevis luks bind -d /dev/vda2 sss '{"t": 2,

"pins": {"tang": [

{"url": "http://tang1.mobile.roninprinciples.com"},{"url": "http://tang2.mobile.roninprinciples.com"},{"url": "http://tang3.mobile.roninprinciples.com"}]

}}'

Quick sample incantation (human readable)

Thank you for attending Ohio Linux Fest!

Portions of the content were based on presentation from:➢ Nathaniel McCallum➢ Brian Atkisson➢ Jim WildmanTechnical references:➢ luks: https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup➢ cryptsetup https://gitlab.com/cryptsetup/cryptestup/➢ Samir’s Secret Sharing:

https://en.wikipedia.org/wiki/Shamir's_Secret_Sharing➢ clevis: https://github.com/latchset/clevis➢ tang: https://github.com/latchset/tang

Resources & Credits