Dr Matt Pritchard, Campus network engineering workshop

19/10/2016 Enabling efficient movement of data into & out of a high-performance analysis environment

JASMIN Data Transfer Zone

Enabling efficient movement of data into & out of a high-performance analysis environment

JISC Workshop:Campus Network Engineering for Data Intensive Science19 October 2016

Dr Matt PritchardCentre for Environmental Data AnalysisSTFC / RAL Space

Outline

• What is JASMIN & who is it for?• Example science use cases & challenges posed• JASMIN “Data Transfer Zone”• Community Organisation• Changing workflows

Context: JASMIN & CEDA

JASMIN infrastructure

• Part data store, part supercomputer, part private cloud…

Science use cases

Testing high resolution climate models

Earthquake monitoring

Whole-mission satellite data processing: greenhouse gases

Air pollution: model-observation comparison

Sea surface temperature from space

Climate data dissemination

JASMIN data growth

Growth of data in the CEDA archive:• Step change in volume with ingestion of

CMIP5• Step change in no of files with ERA40,

EISCAT datasets

• NCEO & NCAS are Atmos / Earth Obs components of CEDA archive

• CMIP & Sentinel data shown separately• Major growth: users exploiting Group

Workspaces

ArchiveGroup Workspaces

gws1 gws2 gws3

NERC Managed Cloud Analysis Compute

NERC Unmanaged Cloud

login bastion transfer

sciscisci Cluster

ArchiveTape

NERC EOS Cloud

ESA CCI SST

ElasticTape

Functional View

JASMIN Internal Network

• Core network designed for optimum performance & flexibility for data analysis– Zero contention, non-blocking– Equal-cost multi-path routing (ECMP)– 3 Terabit/sec internal throughput– Over 1,100 10GbE ports– 12 x 40GbE switches (more than RAL site!)– Compute nodes interchangeable between

virtualization & batch processing

JC2-LSW1 JC2-LSW1 JC2-LSW1JC2-LSW1 JC2-LSW1 JC2-LSW1 JC2-LSW1 JC2-LSW1 JC2-LSW1JC2-LSW1 JC2-LSW1 JC2-LSW1

48 * 16 = 768 10GbE Non-blocking16 x 12 x 40GbE = 192 40GbE ports

S1036 = 32 x 40GbE

JC2-LSW1JC2-LSW1

JC2-SP1 JC2-SP1 JC2-SP1 JC2-SP1 JC2-SP1 JC2-SP1

16 x MSX1024B-1BFS48x10GBE + 12 40 GbE

16 x 12 40GbE = 192 Ports / 32 = 6Total 192 40 GbE Cable

Moving big data

1. DON’T move it!– Co-locate the analysis environment with the data (move the compute)

• Archive data (~5 PB parallel file system mounted RO)• Group Workspaces (~10 PB, same storage technology, mounted RW)• Process the data in-place

2. Coordinate as a community– Reduce N x N transfers– Replicate data to a few key sites

• “Institutional” data transfers– Assign workspace managers

• Get users to talk to each other!

3. End-user data transfers

Community coordination: ESGF

Data transfer improvement

ICNWG progress

• Created “Science DMZ” at core sites– Blueprint

• http://fasterdata.es.net/science-dmz/• Network zone outside site firewall• As close to site border routers as possible• Dedicated suite of high-performance equipment• Include only specific science data transfer services• Exclude all other traffic (general internet usage)• Monitoring/diagnostic tools : perfSONAR

– ICNWG Implementation• Sites benefit from ESnet “guru” expertise & coordination (Eli Dart)

JASMIN Data Transfer ZoneDTZ services:

• CEDA archive FTP (RO)• FTP server

• Xfer servers (RW)• Scp, Rsync• Bbcp• Ssh-based GridFTP

• GridFTP servers (RW)• Cert-based GridFTP• Globus endpoint

• ESGF Data Node (RO)• Cert-based GridFTP• Globus endpoint

Changing workflows

Data “gravity”

Further information

• JASMIN– http://www.jasmin.ac.uk

• Centre for Environmental Data Analysis– http://www.ceda.ac.uk

• STFC Scientific Computing Department– http://www.stfc.ac.uk/SCD/

• JASMIN paperLawrence, B.N. , V.L. Bennett, J. Churchill, M. Juckes, P. Kershaw, S. Pascoe, S. Pepler, M. Pritchard, and A. Stephens. Storing and manipulating environmental big data with JASMIN. Proceedings of IEEE Big Data 2013, p68-75, doi:10.1109/BigData.2013.6691556