Embed Size (px)
Citation preview
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
The ARC data client in EMI-1
Jon Kerr NilsenDept. of Physics, Univ. of Oslo
On behalf of the ARC Data PT
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Why ARC data libs?• ARC data in production• ARC data Architecture• ARC client tools• ARC data in EMI• Plans / way forward
Outline
Jon Kerr Nilsen 2
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Can be used on different platforms (Linux, Windows, Mac, Solaris)
• Can be used from different languages (C++, Python, java)• Pluggable infrastructure which allows for a very limited
set of dependencies from other libraries• Plugins exist for several widely used data transfer and
management protocols (GridFTP, SRM, LFC, https, stdio, …)
• Light-weight and user-friendly, with easy client-side configuration (the default config would work in most cases)
Why use ARC data libs?
Jon Kerr Nilsen 3
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
ARC DDM in production
• ARC is not only for the Nordics
• Highly distributed – e.g. NDGF T-1 is spread over four countries (both CPU and storage)
• Most resources shared with other users/VO’s
Jon Kerr Nilsen 4
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
ARC DDM in production
• 4-6k cores being used • 20-40k jobs per day• Transfering 500MB/s on
average for downloads and uploads –> ~100TB/day
• Approx. half of it on NDGF T-1– Not very big but very
efficient
Jon Kerr Nilsen 5
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
ARC Internals
• Clients can submit jobs using WS interface or to ARC classic and other legacy services via specialized adaptors
• HED handles communication between services and with outside world
• A-REX handles data staging (through libarcdata2) and batch job submission (through LRMS)
• No middleware on compute-nodes
Jon Kerr Nilsen 6
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
libarcdata2
Jon Kerr Nilsen 7
• Generic library for basic data access and movement
• Protocols are supported through plugins (DMCs)
• No external dependencies in libarcdata2, only in plugins
• Same library used on client side and service side – smaller codebase, easier to maintain
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Data client functionality
• Three command line tools– arcls– arccp– arcrm
• Extra functionality in CLI options and URL options
• Focused on what functionality is needed in grid DDM – not on providing a fully POSIX-compliant interface
Jon Kerr Nilsen 8
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Data client functionality
• Protocols are supported through DMC plugins– GridFTP, SRM, LFC, HTTP,
file, stdio
• Clients support whatever protocol with corresponding plugin installed
• All plugins installs via nordugrid-arc-client-tools metapackage
Jon Kerr Nilsen 9
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Documentation
• man arccp• arccp --help
Usage: arccp [OPTION...] source destination
The arccp command copies files to, from and between grid storage elements.
Help Options: -h, -?, --help Show help options
Application Options: -p, --passive use passive transfer (does not work if secure is on, default if secure is not requested) -n, --nopassive do not try to force passive transfer -y, --cache=path path to local cache (use to put file into cache)...
Jon Kerr Nilsen 10
http://www.nordugrid.org/documents/arc-ui.pdf
http://www.nordugrid.org/documents/arc-client-install.html
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Download from LFC – arccp
lfc://lfc1.ndgf.org/grid/test/file.01 ./file.01
• Upload to LFC specifying SRM location and spacetoken– arccp ./file.01
lfc://lfc.ndgf.org/grid/myvo/file.01\ --location=\srm://srm.ndgf.org;spacetoken=MY_TOKEN/test/file.01.a
• List LFC replicas– arcls -L lfc://lfc.ndgf.org/grid/test/file.01
srm://srm.ndgf.org/test/file.01.a:guid=d11c1c69-3400-466c-995e-7d5681a33692
• Remove LFC entry– arcrm lfc://lfc.ndgf.org/grid/test/file.01
Example: LFC
Jon Kerr Nilsen 11
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Copying to/from stdio– arccp gsiftp://grid.uio.no/jobdir/145434643346354/out.txt stdio:///stdouthello
– arccp gsiftp://grid.uio.no/jobdir/145434643346354/out.txt –hello
– arccp - -hellohello
Example: stdio
Jon Kerr Nilsen 12
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• libarcdata2 API available in Python• import arc
from arcom import datapoint_from_urlsrc = datapoint_from_url(turl, ssl_config)dst = datapoint_from_url(fobj.name)mover = arc.DataMover()mover.verbose(verbose)mover.retry(False)mover.passive(True)status = mover.Transfer(src, dst, arc.FileCache(), arc.URLMap())if status != status.Success: raise Exception, ”””Failed to download file. Try again!”””return str(status)
Python API
Jon Kerr Nilsen 13
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• ARC participates in the EMI Data group with two data experts– Zsombor Nagy, Jon Kerr Nilsen
• ARC has no storage element in EMI, limited amount of efforts needed for ARC data libraries so far (need GLUE2 and SRM changes on SEs first)
• Most important task so far: Consolidation
ARC Data in EMI
Jon Kerr Nilsen 14
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Two sets of data clients/libs in EMI– arcdatalib2 and arc data tools– GFAL and lcg_util
• These have overlapping functionality• Important goal for EMI to reduce code
base (and thus reduce maintenance costs)• Need to identify overlap and potential
consolidation possibilities
Data lib consolidation
Jon Kerr Nilsen 15
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• High-level plan– Evaluate current libs/clients– Investigate what features in the libs/clients
are used/needed– Identify scenarios and analyze
• Efforts, costs, benefits, risks, dependencies…
– Suggest a decision for the best solution– Implement this solution
Data lib consolidation
Jon Kerr Nilsen 16
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Had F2F meeting with the gLite and ARC data client developers in March
• A couple of possible consolidation possibilities were found– SRM and LFC libraries
• Also found surprisingly small feature overlap– GFAL focused on POSIX API– ARC data libs focused on grid user needs – functionality
not covered by POSIX API– Both are useful
• More details on EMI all-hands in June :)
Data lib consolidation
Jon Kerr Nilsen 17
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Consolidation, consolidation and consolidation
• Also– Phasing out ng* clients, replaced by arc*
clients– Move from SRM through httpg to SRM
through https– GLUE2 as soon as SEs have implemented– Bugfixing and improvements
Way forward
Jon Kerr Nilsen 18
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Jon Kerr Nilsen 19
Thank you!
Manual: http://www.nordugrid.org/documents/arc-ui.pdfInstallation: http://www.nordugrid.org/documents/arc-client-install.html
Bugs and feature requests: http://bugzilla.nordugrid.org Contact: [email protected]
