Replacing Traditional Backup Systems with ZFSzfs.datto.com/2018_slides/parker.pdf · 2018. 4....

Replacing Traditional Backup Systems with ZFS#ZFS User Conf 2018Calvin Hendryx-Parker@calvinhpSix Feet Up

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

“Backups will be done when it is easy”me

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Back StoryAMANDA 1999-2012Bacula 2007, 2012-2018

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

AMANDAThe Advanced Maryland Automatic Network Disk Archiver

Project started in the early 90sUses native tools (tar, dump)

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

First Love !

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Bacula» Filesystem level backup

» GPLv2 AGPLv3 or Proprietary license

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

“However, if you are new to Unix systems or do not have offsetting experience with a sophisticated backup package, the Bacula project does not recommend using Bacula as it is much more difficult to setup and use than tar or dump.”The Bacula Website

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

The Hidden Cost of Creating Open Source Code 1

“Bacula Systems, went farther and worked with the FSFE to create an agreement whereby Bacula Systems formally states that it will contribute all its development, not bound by a third-party proprietary agreement, to the Bacula community version within a maximum of 5 years delay.”Bacula Blog, December 2013

1 https://blog.bacula.org/110/

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Issues» Difficult to configure

» Many Moving Parts

» Lots of Care and Feeding

» Slow (for our use case)

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

There has to be a

better wayReplacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

What if...Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

there was a filesystem that supported some advanced features» snapshots

» replication

» compression

» data integrity

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

And did them at a Block level?

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

I ❤ ZFSReplacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

FreeBSD(or any OS that supports ZFS) just works™

and we are a FreeBSD shop already!

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Goals1.Reliable

2.Granular

3.Integrity

4.Few Dependencies

5.Easy to Configure

6.Runs with little care and feeding

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Initial Winsfup-director% wc -l bacula-dir.conf 309 bacula-dir.conf

sfup-director% wc -l schedule/schedule.conf 167 schedule/schedule.conf

sfup-director% wc -l clients/jobs.conf 928 clients/jobs.conf

sfup-director% wc -l bacula-fd.conf 27 bacula-fd.confReplacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Per Jail and Bhyve Hostsfup-jail01% grep zfs /etc/periodic.confdaily_zfsnap_enable="YES"daily_zfsnap_recursive_fs="zroot/jails"daily_zfsnap_delete_enable="YES"weekly_zfsnap_enable="YES"weekly_zfsnap_recursive_fs="zroot/jails"weekly_zfsnap_delete_enable="YES"

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Per Trusted Server$ sudo crontab -l -u root# Lines below here are managed by Salt, do not edit@daily /usr/local/sbin/zxfer -dFkv -g 376 -T root@sfupstor01 -R zroot/jails storage/sfup-jail01

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Everything Elsefor quad in ${QUADS[*]}do DAILY=$(ssh $quad zfs list -t snapshot | grep daily-$(date +%Y-%m-%d) | awk '{print $1}' | grep -e "zroot/[a-z]\{3,5\}[-]")

echo $(date) "Running Daily Backup from $quad" >> $LOG for dataset in ${DAILY[*]} do ssh $quad zfs send -R -I base $dataset | zfs receive -vdF storage/backups/bhyve-vms >> $LOG done echo $(date) "Daily Backup Complete" >> $LOG

done

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Bonus!Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp

Questions?

https://github.com/calvinhp/Simple-Reliable-ZFS-Backups-Presentation

calvin@sixfeetup.com@calvinhp

Photo Credits

» https://www.flickr.com/photos/momentsnotice/12889668265

» https://www.flickr.com/photos/tom1231/5055926716

» https://www.flickr.com/photos/63259711@N04/5754479117

» https://www.flickr.com/photos/mio_please/2245667848

» https://www.baculasystems.com/architecture

» https://www.flickr.com/photos/mrpinkeyes/7276108196

» https://www.flickr.com/photos/nsub1/3537823179

» https://www.flickr.com/photos/pmillera4/8538521014

Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp