20
# Daniel Anolik Solutions Architect, Jeppesen Version Your Cloud

Version Your Cloud: Using Perforce to Manage Your Object Storage

Embed Size (px)

DESCRIPTION

Jeppesen uses a combination of their own data centers and a variety of 3rd party cloud providers to support their distribution of navigation data to most of the world’s pilots. But before any data leaves the building it goes through Perforce. Learn why Jeppesen chose to put Perforce at the heart of their global data distribution system, and how you too could leverage Perforce to help provide atomic changesets and true history tracking to your cloud-scale object stores.

Citation preview

Page 1: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

Daniel AnolikSolutions Architect, Jeppesen

Version Your Cloud

Page 2: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

Daniel AnolikSolutions Architect, Airborne Navigation SystemsJeppesen (A Boeing Company)• 8 years with Jeppesen

• 15 years of direct SCM experience, 8 with Perforce

• Software developer, solutions architect, Scrum master, bluegrass mandolin picker, wilderness survival enthusiast…

Page 3: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

Daniel AnolikSolutions Architect, Airborne Navigation SystemsJeppesen (A Boeing Company)• 8 years with Jeppesen

• 15 years of direct SCM experience, 8 with Perforce

• Software developer, solutions architect, Scrum master, bluegrass mandolin picker, wilderness survival enthusiast,

and lucky father of this baby girl:

Page 4: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

Jeppesen at a Glance• 80-year strong company founded in aviation safety• Wide array of software supporting:

– airborne navigation systems– flight planning services– pilot training – airline operations management

Page 5: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

Supplier of Critical Aviation Data

Page 6: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

• Distribute many data types to many applications running on many platforms

• Support frequent data updates for hundreds of thousands of customers around the world

• Transfer hundreds of terabytes of data each month• Don’t be a bottleneck!

The Challenge

Page 7: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

Initial Plan

Data Brokering Magic

Jeppesen DataProduction

Magic

JeppesenInfrastructure CDM

Cloud Object

Storage

Workflow Magic

Data PublishingInterface

Boring File Server

Page 8: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

The system must support:• Fine read/write access controls within storage tree

• Atomic change-sets for publishing files

• Detailed history tracking

• Individual file restores

• Data lifecycle management

Hmm…It’s starting to sound like we need a real CMS system in there

Pesky Implementation Constraints

Page 9: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

Improved Plan

Data Brokering Magic

Jeppesen DataProduction

Magic

JeppesenInfrastructure CDM

Cloud Object

Storage

Workflow Magic

Data PublishingInterface

Page 10: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

• Supported all our implementation constraints with minimal customization

• Leverage existing in-house expertise

• P4D had very stable track record at Jeppesen

• Efficient, scalable blob storage

Why Perforce vs Other CMS?

Page 11: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

• Keeping changes atomic:– File sets are submitted to P4 as single changelist– Data is processed by the changelist– Data is published by the changelist

• Files are never ‘available’ to our applications until all files in a given changelist are available

Leveraging Perforce: Changelists

Page 12: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

• Changelists tell us:– What files were published,– when,– and by whom

• Spec depots!• Soft deletes of files minimize ‘clutter’ while still

supporting data retention requirements

Leveraging Perforce: History

Page 13: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

• Custom counters– are defined for each data type

– track last successful changelist processed

– provide a central, durable mechanism for state tracking

across different work nodes in the system

Leveraging Perforce: Counters

Page 14: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

• Job queue are simple files in p4

• In process jobs are ‘locked’

• Completed jobs are ‘deleted’

• Full job history is retained

Leveraging Perforce: Locks

Page 15: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

Leveraging Perforce: Branching

Private “Staged” Data

Public “Promoted” Data

p4 integrate

Depot paths represent the promotional model of the data

Page 16: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

• P4 protections restrict access by:– depot path/data library– pattern matching/file extensions – IP addresses

• Service accounts granted read and write access to only what they need

Leveraging Perforce: Security

Page 17: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

• 100% of interaction with Perforce fully automated through the P4Java API

• Custom utilities used to:– Read and write data in depot

– Manage branch integrations

– Read and update counters

– Read changelist details

– ‘Expire’ content

Leveraging Perforce: Java API

Page 18: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

All the great capabilities of cloud based object storage, plus true configuration management:

– Detailed history tracking and extreme auditability

– Reproducibility: roll back any part of your ‘cloud’ to any point in time

– Atomic changelist management of file sets

The Result: Object Storage++

Page 19: Version Your Cloud: Using Perforce to Manage Your Object Storage

#

Any Questions?

#

Page 20: Version Your Cloud: Using Perforce to Manage Your Object Storage

##

Thank you!And have a safe flight home.

Daniel [email protected]