The BlackBox projectSafely storing secrets and credentials in Git
for use by Puppet
Tom Limoncelli, SRE, StackExchange.comBlog: EverythingSysadmin.com
125+ Q&A CommunitiesServerFault.comStackOverflow.com
(We <3 Puppet!)
StackExchange.com
What are secrets?
Anything you don’t want exposed externally.
● SSL Certificates (the private bits)● Passwords● API keys
Puppet manages secrets
If you store secrets in git, you’re gonna have a bad time.
● Laptops get stolen.● Workstations have guest accounts● “Circle of Trust” now includes:
○ Everyone with admin access to workstations.■ Your desktop support people?
○ Everyone with admin access to your git server:■ Server team, storage team, backup team
○ Everyone you collaborate with that wants read-only access to Puppet manifests.
You have 3 bad options:
1. Deny git access. (Hurts collaboration)2. Permit git access. (Hurts security)3. Email individual files. (Hurts… just hurts)
Option 4: Encrypt secret parts
● If a file contains secrets, encrypt before checking into Git.
● Need to edit a secret?○ Decrypt - Edit - Encrypt
What about Puppet master?
● After “git pull”, decrypt all files.○ Automate this as part of CI.
● Files are unencrypted “at rest”.● This does not decrease security:
○ No worse than what we were doing before.○ If you can break into root or puppet on the master,
you’ve already won.
Easy, right?
Decrypt:gpg -q --decrypt -o secret.crt secret.crt.gpg
Encrypt:gpg --yes --trust-model=always --encrypt -o secret.crt.gpg $(<keynames) secret.crt
Easy, right?
Decrypt:gpg -q --decrypt -o secret.crt secret.crt.gpg
Encrypt:gpg --yes --trust-model=always --encrypt -o secret.crt.gpg $(<keynames) secret.crt
● ...and don’t make any typos when entering the command● ...and don't accidentally check in the unencrypted version
Security is 1% technology plus 99% following the procedures correctly.
Any process with more than 1 step probably won't be followed consistently most of the time.
Related reading: "Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0”, Alma Whitten", Usenix Security 1999
Therefore…. we automate
Introducing: Blackbox
Scripts for keeping Puppet secrets in git/hg.
User commands:
Decrypt for editing:blackbox_edit_start.sh file
Encrypt when done:blackbox_edit_end.sh file
First time a file is encrypted:
Enroll a file into the system:blackbox_register_new_file.sh file
Commands that act on all GPG files:
Decrypt all files: (for use on puppet master)blackbox_postdeploy.sh
Re-encrypt all files: (after new users added)blackbox_update_all_files.sh
Everyone has their own key
This doesn’t use “symmetric encryption” where there is one passphrase to decrypt/encrypt all files.
We maintain a keyring of:● Each person that should have access.● A key for the Puppet master.
Indoctrinate a new user:
1. New user does this:
● Create GPG key.● Add their username@host to blackbox-admins.
txt● git commit -a
(Currently a doc, not a script. Patches gladly accepted.)
Indoctrinate a new user:
2. Existing admin does this:
$ gpg --import keyrings/live/pubring.gpg
$ blackbox_update_all_files.sh
$ git commit -a
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Code is open source as of TODAY
● Entirely written in bash.● MIT License.● Download it now:
○ https://github.com/StackExchange/blackbox
In the project’s first 9 months:
StackExchange/ServerFault has eliminated plaintext secrets in our Puppet git repo.● 7 SREs+Devs sharing the repo securely.● 50+ files now stored encrypted.
○ Mostly SSL certs and SSH private keys.● 40+ individual passwords/API keys:
○ Everything from SNMP communities, SaaS API keys, and many many passwords.
Future plans
❏ Open source scripts.❏ More usability enhancements.❏ Better setup documentation.
Join the open source projecthttp://github.com/StackExchange/blackbox
Q&A
URLs from this talk:https://github.com/StackExchange/blackbox
EverythingSysadmin.com
Shameless plugPre-order now! Save 35%
Ships in September.
informit.com/TPOSADiscount code TPOSA35
Read “rough cuts” today:safaribooksonline.com
Q&A
URLs from this talk:https://github.com/StackExchange/blackbox
EverythingSysadmin.cominformit.com/TPOSA (code TPOSA35)
● Easier transition. No Puppet code changes for big files like SSL certs.
● Faster. Zero run-time performance impact on master.
● eyaml didn’t exist when we started.
Why didn’t we use eyaml?