Embed Size (px)
Citation preview
Replacing Traditional Backup Systems with ZFS#ZFS User Conf 2018Calvin Hendryx-Parker@calvinhpSix Feet Up
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
“Backups will be done when it is easy”me
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Back StoryAMANDA 1999-2012Bacula 2007, 2012-2018
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
AMANDAThe Advanced Maryland Automatic Network Disk Archiver
Project started in the early 90sUses native tools (tar, dump)
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
First Love !
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Bacula» Filesystem level backup
» GPLv2 AGPLv3 or Proprietary license
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
“However, if you are new to Unix systems or do not have offsetting experience with a sophisticated backup package, the Bacula project does not recommend using Bacula as it is much more difficult to setup and use than tar or dump.”The Bacula Website
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
The Hidden Cost of Creating Open Source Code 1
“Bacula Systems, went farther and worked with the FSFE to create an agreement whereby Bacula Systems formally states that it will contribute all its development, not bound by a third-party proprietary agreement, to the Bacula community version within a maximum of 5 years delay.”Bacula Blog, December 2013
1 https://blog.bacula.org/110/
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Issues» Difficult to configure
» Many Moving Parts
» Lots of Care and Feeding
» Slow (for our use case)
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
There has to be a
better wayReplacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
What if...Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
there was a filesystem that supported some advanced features» snapshots
» replication
» compression
» data integrity
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
And did them at a Block level?
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
I ❤ ZFSReplacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
FreeBSD(or any OS that supports ZFS) just works™
and we are a FreeBSD shop already!
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Goals1.Reliable
2.Granular
3.Integrity
4.Few Dependencies
5.Easy to Configure
6.Runs with little care and feeding
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Initial Winsfup-director% wc -l bacula-dir.conf 309 bacula-dir.conf
sfup-director% wc -l schedule/schedule.conf 167 schedule/schedule.conf
sfup-director% wc -l clients/jobs.conf 928 clients/jobs.conf
sfup-director% wc -l bacula-fd.conf 27 bacula-fd.confReplacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Per Jail and Bhyve Hostsfup-jail01% grep zfs /etc/periodic.confdaily_zfsnap_enable="YES"daily_zfsnap_recursive_fs="zroot/jails"daily_zfsnap_delete_enable="YES"weekly_zfsnap_enable="YES"weekly_zfsnap_recursive_fs="zroot/jails"weekly_zfsnap_delete_enable="YES"
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Per Trusted Server$ sudo crontab -l -u root# Lines below here are managed by Salt, do not edit@daily /usr/local/sbin/zxfer -dFkv -g 376 -T root@sfupstor01 -R zroot/jails storage/sfup-jail01
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Everything Elsefor quad in ${QUADS[*]}do DAILY=$(ssh $quad zfs list -t snapshot | grep daily-$(date +%Y-%m-%d) | awk '{print $1}' | grep -e "zroot/[a-z]\{3,5\}[-]")
echo $(date) "Running Daily Backup from $quad" >> $LOG for dataset in ${DAILY[*]} do ssh $quad zfs send -R -I base $dataset | zfs receive -vdF storage/backups/bhyve-vms >> $LOG done echo $(date) "Daily Backup Complete" >> $LOG
done
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Bonus!Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
Questions?
https://github.com/calvinhp/Simple-Reliable-ZFS-Backups-Presentation
[email protected]@calvinhp
Photo Credits
» https://www.flickr.com/photos/momentsnotice/12889668265
» https://www.flickr.com/photos/tom1231/5055926716
» https://www.flickr.com/photos/63259711@N04/5754479117
» https://www.flickr.com/photos/mio_please/2245667848
» https://www.baculasystems.com/architecture
» https://www.flickr.com/photos/mrpinkeyes/7276108196
» https://www.flickr.com/photos/nsub1/3537823179
» https://www.flickr.com/photos/pmillera4/8538521014
Replacing Traditional Backup Systems with ZFS — #ZFS User Conf 2018 — @calvinhp
