Open Source Backup Conference 2014: Bareos in scientific environments, by Jan Behrend

für Radioastronomie

Max-Planck-Institut

Bareos in Radio Astronomy - Scaling up using Virtual Full Backups

Bareos in Radio Astronomy –Scaling up using Virtual Full Backups

Jan Behrend

Max Planck Institute for Radio Astronomy

Open Source Backup ConferenceSeptember 23rd 2014


Max-Planck-Institut


OverviewAbout the InstituteBackup Goals and LimitationsThe ChallengeImplementationConfiguraton StrategyVirtual Full BackupsIntegration with DRBDIntegration with REARWishlist


Max Planck Institute for Radio Astronomy

– Founded in 1966– Main Building Bonn / 100m Radio Telescope Effelsberg– Apex Telescope in the Atacama Desert, Chile

– IT staff of 11 people– 204 servers (132 virtualized, 110 Bareos clients)– 64 node cluster– . . .

http://mpifr.de


Max-Planck-Institut


Lofar Antenna Field

“Software” Telescope44 Stations


Max-Planck-Institut


Scientific Raw Data

Project Output per Observation RunK-Band receiver: 2 TiBH1-Survey: 3 TiBPulsar search: 4 TiBLeap: 7 TiBLofar: 50 TiBGlow: 120 TiB. . .

∑186 TiB

Forget it!


Max-Planck-Institut


Scientific Raw Data

Project Output per Observation RunK-Band receiver: 2 TiBH1-Survey: 3 TiBPulsar search: 4 TiBLeap: 7 TiBLofar: 50 TiBGlow: 120 TiB. . .

∑186 TiB

Forget it!


Max-Planck-Institut


GoalsFast recovery of:

specific files / directories(many) complete systems

LimitationsTime

Backup window (scientists are night birds)Network bandwidth (usually 1GBit/s)Resources on backup clients (I/O, RAM, CPU)

Storage space (disk and tape)

What can we do?parallel jobs, Virtual Full Backupsvolume retention


Max-Planck-Institut




LimitationsTime





Max-Planck-Institut




LimitationsTime





Max-Planck-Institut


“The Remains of the Day” without Raw Data

Full Backup Volume: 55 TiB (1.5M files / TiB)Differential Backup Volume: 566 GiB (4K files / GiB)Incremental Backup Volume: 102 GiBTwo backup copies

Time for a complete Full Backup and its 2nd copy

55 TiB130 MiB

s

· 2 · 2 · 1 d60 · 60 · 24 s

= 20.54 days

High Volume “Longterm” ClientsVirtual Fulls every half year, Incrementals every SaturdayLongterm Full Backup Volume: - 50 TiB (1M files / TiB)⇒ Full Volume every 8 weeks: 5 TiB (6M files / TiB)


Max-Planck-Institut





55 TiB130 MiB

s

· 2 · 2 · 1 d60 · 60 · 24 s

= 20.54 days



Max-Planck-Institut





55 TiB130 MiB

s

· 2

· 2 · 1 d60 · 60 · 24 s

= 20.54 days



Max-Planck-Institut





55 TiB130 MiB

s

· 2 · 2

· 1 d60 · 60 · 24 s

= 20.54 days



Max-Planck-Institut





55 TiB130 MiB

s

· 2 · 2 · 1 d60 · 60 · 24 s

= 20.54 days



Max-Planck-Institut





55 TiB130 MiB

s

· 2 · 2 · 1 d60 · 60 · 24 s

= 20.54 days



Max-Planck-Institut


Backup Hardware2 Tape Libraries: Spectra Logic T950

2 x LTO5 drives80 Slots a 1,5 TiB = 160 TiB (too small by now)Dedicated cleaning partition

2 physical servers (Fujitsu RX300S6)2x 30 TiB RAID storage (Nexan Satabeast2)(Mostly) dedicated 1GbE/10GbE backup network


Max-Planck-Institut


Hardware Infrastructure

FC SAN FC SAN

BACKUPSRV2cold standby

BACKUPSRV1

BACKUP LAN BACKUP LAN

INTRANET BONNINTRANET EFFELSBERG

TAPE LIBRARY 1TAPE LIBRARY 2

8GBit

1GBit

3x10GBit

10GBit 10GBit

SATABEAST (30 TiB)

40km

SATABEAST (30 TiB)


Max-Planck-Institut


Configuration StrategyIncrementals to disk for better restore performanceFulls and Differentials to cheaper tape storagePer client config (DIR and SD) for easier configmanagement:

compression, quota, reservation, encryption ...easy templating for new backup clients

Focus on schedules / retentionDaily backup summary (3rd party tools)Dedicated DB partition· · ·


Max-Planck-Institut


Virtual Full Backups

Reorganizes existing backups to create a new Full

Last Full (2 months ago)FullPool

Last Differential (1 week ago)DiffPool

Incrementals (x days ago)IncrementalPool

New Virtual Full (today)VirtualPool

Subsequent Migration


Max-Planck-Institut


Virtual Full Backups

+++ No backup client interaction!(basically) no backup time limit

– Needs designated pool to avoid lockups– No “MaxFullInterval” config option


Max-Planck-Institut


What is DRBD?

Taken from http://www.drbd.org/home/what-is-drbd


Max-Planck-Institut


Integrating Bareos with HA / DRBD

DRBD volume backed up by HA controlled BAREOS instance

IMAP FailoverIP: 134.104.18.110BAREOS on /REAR enabled

IMAP FailoverIP: 134.104.18.111BAREOS on /REAR enabled

/var/mail on DRBDBAREOS on /var/mailIP: 134.104.18.77...

Client {Address = 134.104.18.110...FileSet {File = /

Client {Address = 134.104.18.77...FileSet {File = /var/mail

Client {Address = 134.104.18.111...FileSet {File = /


Max-Planck-Institut


REAR: Relax and Recover

Linux bare metal disaster recovery solutionCreates bootable mediaRecovery takes care of all disk configurationCan use Bareos to restore content


Max-Planck-Institut


Bareos Client config

/etc/rear/local.confGRUB_RESCUE=OUTPUT=ISOBACKUP=BAREOSBACKUP_URL="rsync://backupsrv/srv/rear/"

/etc/bareos/bconsole.confConsole {

Name = client-restorePassword = "password"

}


Max-Planck-Institut


Bareos Client config

/etc/rear/local.confGRUB_RESCUE=OUTPUT=ISOBACKUP=BAREOSBACKUP_URL="rsync://backupsrv/srv/rear/"

/etc/bareos/bconsole.confConsole {

Name = client-restorePassword = "password"

}


Max-Planck-Institut


Bareos Server config

/etc/bareos/conf.d/client.dir.confConsole {Name = client-restorePassword = "password"ClientACL = client-fdJobACL = Restore, clientFileSetACL = clientCommandACL = status, restore[...]

}

http://relax-and-recover.org


Max-Planck-Institut


WishlistIntegration of virtualization solutions on hypervisor levelWrite to more than one destination at onceRolling Spool/DespoolAutomatic spooling when writing to tapeNo copy of jobs using a base job?”MaxFullInterval“ for Virtual Full Backups


Max-Planck-Institut


Want configs? Send me a mail to [email protected]

Any questions?

Notable reference:http://myunix.dk/category/bacula


Max-Planck-Institut


Want configs? Send me a mail to [email protected]

Any questions?

Notable reference:http://myunix.dk/category/bacula