View
218
Download
0
Category
Tags:
Preview:
Citation preview
The Atlas Petabyte Datastore
A grid enabled, networked data storage system:
CrystalGrid Workshop 15th Sept 2004
David Corney. d.r.corney@rl.ac.uk
Data Store Overview
• General purpose, multi user, data archive. • In use over 20 years. Four major upgrades.• Current capacity 1PB – largest (non dedicated) multi user
system in UK academia?• Grid Interfaces:
– SE (Storage Element) – will be SRM compliant– SRB interface (Storage Resource Broker)
STK 9310
Switch_1 Switch_2
RS6000 RS6000RS6000 RS6000
fsc0 fsc1 fsc1fsc0
9940B 9940B 9940B 9940B 9940B 9940B 9940B 9940B
1 2 3 4 5 6 7 8
11 14 11 1415
fsc1fsc0fsc1fsc0
12 13 12 13 15
rmt1 rmt4rmt3rmt2rmt5-8 rmt5-8rmt5-8rmt5-8
A A A A A A A A
STK 9310 “Powder Horn”
Gbit network
1.2TB 1.2TB 1.2TB 1.2TB
flfstk
tapeserv
Farm Server
flfsys(+libflf)
user
flfscan
data transfer (libvtp)
catalogue data
STK tape drive
cellmgr
Catalogue Server (brian)
flfdoexp(+libflf)
flfdoback(+libflf)
datastore (script)
Robot Server (buxton)
ACSLS
API
control info(mount/dismount)
data
data
Tape Robot
flfsys user commands (sysreq)
SE
recycling (+libflf)
read
read
read
Atlas Datastore Architecture
28 Feb 03 - 2 B Strong
SSI
CSI
flfsys farm commands (sysreq)
LMU
flfsys admin commands(sysreq)
administrators
flfaio
flfaio
flfaio
IBM tape drive
flfqryoff(copy of
flfsyscode)
Backupcatalogue
stats
flfsys tapecommands
(sysreq)
servesys
pathtape
long name(sysreq)
short name(sysreq)
frontend
backendPathtape Server (rusty)
(sysreq)
importexport
flfsys import/export commands (sysreq)
libvtp User Node
I/E Server(dylan)
?
Copy BCopy C
ACSLS
cache disk
Copy A
vtp
vtp
user program
tape
(sysreq)
Strategy -
• De-couple user and application from storage media. • Upgrades and media migration occur “behind the scenes”• High resilience - very few Single Point Failures• High reliability high, availability (99.9986% in 2003)• Constant environmental monitoring linked to alarm/call out• Lifetime data integrity checks hardware and software• Fire safe and off-site backups; Tested disaster recovery
procedures; media migration, recycling• Easy to exploit (endless) new technology• Technology watch to monitor future technology path
Robot History
• M860 – 110GB
• STK 4400– 1.2Tbytes
• IBM 3494– 30Tbytes
• STK 9310– 1Pbyte
Hardware upgrade - completed Jun 2003
• STK 9310 “Powderhorn” with 6000 slots (1.2Pbytes)• 4 IBM 3590 B drives now phased out
– 10 Gbyte native– 10 Mbyte/s transfer
• 8 New STK 9940B drives– 200 Gbyte native– 30Mbytes/sec/drive transfer – 240Mbyte/sec theoretical maximum bandwidth
• 4 RS6000 Data servers (+ 4 “others”)• 1Gbit networking (Expected to become 10Gbit by 2005)• Data Migration to new media completed ~ Feb 2004
Users
• Particle Physics Community (LHC: CMS, Atlas, LHcb,….)• ISIS, British Atmospheric Data Centre• EISCAT (Radar research)• National Earth Observation Data Centre• World Data Centre, BITD• Central Laser Facility• Diamond…• National Crystallography Service, Southampton University, • WASP, VIRGO Consortium• Integrative Biology,• Others…
Interfaces
• “Light weight” interfaces:– Client server configuration:
• “tape” command for many platforms• Virtual Tape Protocol (VTP)• Fortran and C callable library
• “Heavy weight” interfaces:– SRB interface– SE interface developed for EDG/GRIDPP/GRIDPP2…
SRB-ADS architecture
SRB MCATDatabase
SRB MCATServer
SRB ADSServer
SRBClient
SRB DiskServer (Local Server)
Atlas Data Store SRB ADS Server
SRB-ISIS server instance
SRB-BADC server instance
SRB-CCLRC server instance
Port 5600
Port 5601
Port 5602
Adding Interfaces for ADS
ADS CentralCatalog Serverflfsys
ADS FarmServer
ADS FarmServer
ADS FarmServer
SRB01Server
SRB02Server
ADS FarmServer
VTP InterfaceSRB Interface
LCGServers
SE Interface
SRBUsers
SEUsers
VTPUsers(tape)
Logical Resource for Containers
ADS-cacheresource
ADS-taperesource
ADS-logical-resource
Sput –c <container-name> <data-file>
Ssyncont
Copies container from cache to tape
The Storage Element (SE) mass storage interface
The Storage Element (SE)• A component of European Data Grid (EDG/EGEE) middleware
developed by CCLRC’s e-Science & PPD departments
• Uniform Grid Interface which enables a standard protocol for mass data transfer across the grid, between the many diverse Mass Storage Systems, including:
– Atlas Petabyte Data Store
– CASTOR
– ENSTOR
– HPSS
– Others…
SE Deployment
CERN – Castor and disk
UAB Barcelona – Castor
RAL – Atlas DataStore and Disk
ESA/ESRIN – disk
CC-IN2P3 – HPSS
INFN / CNAF – disk
FZK Karlsruhe – disk
Questions?
Digital Curation Centre
• Joint collaboration between CCLRC, UKOLN, and Edinburgh and Glasgow Universities.
• Provide advice, support, research and Development into aspects of Digital Curation for the UK HE community
• Funded jointly by JISC and EPSRC - £1m/year for three years initially. Feb 2004- 2007
• Establish collaboration with industrial partners…
Objectives
• Vibrant research programme
– addressing the wider issues of digital curation
• Collaborative Associates Network of Data Organisations
– strong links across existing community of practice
– engagement with curators (individuals & organisations)
• Services
– to evaluate tools, methods, standards and policies
– a repository of tools and technical information
• ‘Virtuous circle’
– expertise, experience & requirement feed into the DCC research programme
Digital Curation Centre - Organisation
Service Operations Group
management & governance
Industryresearch collaborators
standards bodies
users: communities of practice
U. of EdinburghU. of Glasgow
UKOLN(Bath)
NDCC/NeSC focus & physical presence
curation organisations e.g. DPC
JISC & Research Councils
Management BoardAdvisory
Group
Collaborative Associates Network of DataOrganisations
CCLRC
Steering & Policy Committee
Research Co-ordination Committee
CCLRC’s role within the DCC
• Standards watch
• Standards definition and publication
• Tools watch
• Tools selection and certification
• Registry of metadata standards
• Metadata research
DCC role in Certification
• DCC will help to create
– Standards against which to perform audit and certification• OAIS Reference Model and follow-on work
– Processes for accreditation and certification• Work in Digital Repository Certification Task Force
– Organisation(s) to perform accreditation and certification
ADS SRB Interface for CMS
SRBMCATserver
SRBADS
Server
ADS ServerFarm
Oracle Database Cluster
ADSTape Robot
System
ADSPathtape
server
File System
File System
SRBClient
SRBFile
Server
SRBClient
SRBFile
Server
CSF disk serverAt RAL
CMS nodeExternal to RAL
SRBClient,
External
Recommended