Upload
amazon-web-services
View
533
Download
0
Embed Size (px)
Citation preview
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Colin Lazier, General Manager Amazon Glacier, AWS Snowball, & AWS Snowmobile
Jay Littlepage, VP, Infrastructure & Operations, DigitalGlobe
Rich Ridolfo, Sr. Dir, Operations, Philips
December 1, 2016
NEW LAUNCH!
IntroducingAWS Snowmobile and AWS Snowball Edge
Migrating large scale data is challenging
Speed
$ Cost
Logistics
Security
The AWS data transport solution
AWS Snowball—petabyte scale data transport
• Rugged 8.5G impact case
• E-ink shipping label
• Data encryption end-to-end
• 50 TB capacity/10G network
E-ink shipping
label
RE:INVENT 2015 LAUNCH
How Snowball moves data into and out of AWS
Create
a job
Connect the
Snowball
Copy data to
the Snowball
Your data
moved to S3
In transit to you Delivered to you Delivered to AWS At AWS
Job created Job completed
Snowball to date
SPEED
COSTLOGISTICS
SECURITY • 80TB Snowball
• HIPAA Compliance
• Regions:
N. Virginia, Ohio, Oregon,
N. California, GovCoud,
Ireland, Frankfurt, Sydney
• Client updates
• HDFS Ingest
• S3 Compatible Endpoint
100x
Challenges at
extreme
scale
80TB
100PB
Exabyte
Increased logistics complexity Increased time for cycles
Snowmobile broad strokes
• 100 Petabytes per job
• Truck with 45 foot container
• 68,000 pounds
• Up to 500Gbps
• Site Survey
• Dispatch
• Fill
• Return for Ingest
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jay Littlepage
DigitalGlobe
Q3 2016 IT All Hands - August 2016
Worldview3 location: 1480 km west of Los Angeles (Lat/Lon 32.58 N/133.96 W)
slant range from Las Vegas: 1950 km at an azimuth of 263 deg (W)
Worldview3 location: 1480 km west of Los Angeles (Lat/Lon 32.58 N/133.96 W)
slant range from Las Vegas: 1950 km at an azimuth of 263 deg (W)
Dealing with “Data Boulders from Space”
• A single image from
WorldView3 is 12GB.
• We are imaging
24x7, adding 85TB
of imagery to our
archive, every day.
0
200
400
600
800
1,000
1,200
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Are
a C
ollecti
on
Per
Year
(Millio
ns S
QK
M)
Year
WV3
WV2
WV1
GE1
QB
IKONOS
“Seeing a Better World” for 16 years, with more sensors at higher resolution,
adds up to a data management challenge.
Release the images!
How? Snowball!
$Tape
Archive
DMP
Service Nodes
First stage booster: push interesting data sets to AWS via
Snowball
$$
Staging
Cache
Fast
Cache
$$$
. . . DMP
Data Movers
Image Processing
Client
Data
Mover
Where?/fast/cache/location
Client
Clearly, we needed a bigger
Snowball. One visible from space.
DMP
Service Nodes
Second stage booster: push entire wideband archive to Glacier vault
via Snowmobile. No more tape robots!
$$
Staging
Cache
Fast
Cache
$$$
. . . DMP
Data Movers
Image Processing
Client
Data
Mover
Where?/fast/cache/location
Data
Mover
Snowmobile
Lessons we are learning with Snowmobile
• This is not plug and play (yet)
• Pre-planning and coordination is
critical.
• Understand payloads
beforehand – a lot of tuning to
optimized for our large files (i.e.
data boulders from space)
• The secret sauce: Compute
intensive translation between
legacy file systems like ours and
the data center on wheels
On orbit: Cloud-centric data management.
Where?
https://s3.amazon.com/bucket/obj
Location
Service / Stage
Instances
Where?
https://s3.amazon.com/bucket/obj
Location
Service / Stage
Instances
DMP
Service Nodes
Fast
Cache
$$$
Image Processing
Client
Where? /fast/cache/location
Learn more @ http://blog.digitalglobe.com/2016/12/01/hauling-data-boulders-from-space/
What can we do with our data freed from the dungeon?
Collect data Create job Copy data Moved to S3
Challenges at the EdgeM
IGR
AT
ION
CO
LL
EC
TIO
N
Create job Copy data Moved to S3
Snowball Edge broad strokes
Protected
and stored
CO
LL
EC
TIO
N
Accessible and
consumable
Short
time frame
Migrate to
the cloud
Snowball Edge broad strokes
Lambda function
Greengrass
integration
CloudIoT
Device
Snowball
clusters
New interfaces (HW)
and Software (S3 & NFS)
Snowball Edge Key Features
Extension of your
data center
Process data Expedites
move
Encrypted, secure,
and embedded
compute
Write data directly
as data is generated
Offers a fast and
cost effective way to
ensure data can be
quickly transferred to
and from the cloud
Simplifies
data transfer
Uses standard
and familiar tools
for the data
transfer process
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rich Ridolfo, Philips Healthcare
Philips HealthSuite & AWS
Philips HealthSuite & AWS
• What is HealthSuite?
• What are the challenges of IT in the Clinical Healthcare realm?
• Snowball Edge in HealthSuite
30
Clinical Healthcare IT Challenges
• Reliability / Availability / Serviceability – Always Up
• Performance – 3 seconds or less [500MB – 2TB per object]
• Connectivity – Capacity and Reliability
• User Perception / Trust
Mitigation – On-site edge device
31
Technology is now key to the delivery of care
Connectivity
Lack of access to broadband [ 25Mbps down / 3Mbps up ] affects:
• 10% of all Americans ( 34 million people )
• 39% of rural Americans ( 23 million people )
• 41% of Americans living on Tribal lands ( 1.6 million people)
• 68% living in rural areas of Tribal lands ( 1.3 million people )
• 66% of Americans living on U.S. territories ( 2.6 million people )
• 98% living in rural areas territorial areas (1.1 million people )
Source: https://www.fcc.gov/reports-research/reports/broadband-progress-reports/2016-broadband-progress-report
Broadband internet access remains a challenge through the US
Desired Solution Characteristics
Survivability / Severability
Universal Codebase
End-to-end Security
Redundancy
Scalability
Single point of management
33
IntelliSpace Console Critical Care
Patient Centered ICU Dashboard
FDA Class 2 Medical Device
• Visualization of data from many sources
• Clinical Decision Support & Error Prevention.
• Dynamic, Task Specific Data Displays
• Checklists & Smart Alerts
• Collaborative Workspace – Care Team, Patient & Family.
What is it?
IntelliSpace Console Critical Care
Aggregate over 1200 ICU patient data points per day
• Vital Signs
• Labs
Process – Compute – Predict
Present
Snowball Edge
Storage + Compute
Encrypted
Tamperproof
Redundant
Philips IntelliSpace Console Critical Care
HealthSuite + AWS Snowball Edge
35
Snowball Edge – Compute
• Running IntelliSpace Console Critical Care
Execute for data transformation
• Real-time data analysis and results delivery
• Site Severability
Snowball Edge – Store
• DB Host
• Disk encrypted - security
The Solution
36
Snowball Edge
On-site hub for communication, data aggregation & pre-processing.
Pathway
s
Analytics
Getting started
Learn more about Snowmobile and Snowball Edge
https://aws.amazon.
com/snowmobile
Contact
AWS Sales
Go to the
management
console
https://aws.amazon.
com/snowballedge
Snowmobile Snowball Edge
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage “Office Hours”Meet the People who Build AWS Storage
Who: Lead Software Development Engineers, Architects, and Technical PMs
Where: Storage Booth Walk-up Bar
When: Exhibit hours (Tues 5-7pm, Wed & Thurs 10:30a-6:00p)
What: Architecture best practices, code reviews, feature requests
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Questions & Answers
Thank you!
Batches and Streams
Direct
Connect
Snowball,
Snowball Edge,
Snowmobile
3rd Party
Connectors
Transfer
Acceleration
Storage
GatewayKinesis Firehose
File
Amazon EFS
Block
Amazon EBS (persistent)
Object
Amazon
GlacierAmazon S3 Amazon EC2
Instance Store (ephemeral)