Upload
norah-morgan
View
222
Download
3
Embed Size (px)
Citation preview
VDT 1
The Virtual Data Toolkit
Todd Tannenbaum(Alain Roy)
VDT 2
What is the VDT?
• A packaging of software– Grid software (Globus, Condor-G…)– Virtual data software (Chimera)– Utilities
• An easy installation mechanism• Testing and hardening• Support
VDT 3
Who makes the VDT?
• Grid Physics Network (GriPhyN)– Constructs the VDT
• International Virtual Data Grid Laboratory (IVDGL)– Testing and hardening
Very tight collaboration between GriPhyN and IVDGL
VDT 4
Who makes the VDT? (2)
• Core VDT Team:– Miron Livny: The boss– Alain Roy– Carey Kireyev
• VDT Testing– Xin Zhao– Brian Moe
• Pacman– Saul Youssef
VDT 5
Who uses the VDT?
• GriPhyN collaborators– USCMS: In use today– USAtlas: In use today– LIGO: Will use soon– SDSS: Will use soon
• European Data Grid– Uses subset of software– Uses just RPMs
• LCG
VDT 6
What exactly is in VDT?
• VDT 1.1.8:– Globus 2.2.4 + advisories + patches– Condor & Condor-G 6.5.1– Chimera/Pegasus– RLS– GLUE Schema– CA Certificates– Fault Tolerant Shell– EDG’s Make Gridmap– EDG’s CRL Update– ClassAds– Netlogger
VDT 7
What exactly is in VDT?
• VDT 1.1.8:– Globus 2.2.4 + advisories + patches– Condor & Condor-G 6.5.1– Chimera/Pegasus– RLS– GLUE Schema– CA Certificates– Fault Tolerant Shell– EDG’s Make Gridmap– EDG’s CRL Update– ClassAds– Netlogger
VDT 8
Grid Software Installation
Typical Grid SoftwareInstallation Experience…
VDT Installation Experience!
VDT 9
VDT Installation
• 2 Methods– Pacman– RPM
VDT 10
Pacman Installation
• Goal: – Type a single command– Everything downloads– Everything installs– Everything is configured– No questions asked
• We’re close:– A few questions if you’re root– Basic configuration, may need changing
VDT 11
Pacman Installation (2)
• Download Pacman– http://physics.bu.edu/~youssef/pacman/
• Install VDT– cd <install-directory>– pacman -get VDT-Server– pacman -get VDT-Client– ls
condor/ globus/ post-install/ setup.sh
edg/ gpt/ replica/ vdt/
ftsh/ perl/ setup.csh vdt-install.log
• Use
VDT 12
Pacman post-installation
• Post-install directory:– Notes on configuration choices made– Instructions for editing configuration
• Configuration scripts:– Globus configuration– Condor configuration
VDT 13
RPM Installation
• Subset of whole VDT– Globus– Condor-G
• Nice RPMs:– We repackage Globus– A dozen Globus RPMs, not hundreds
• No configuration• No post-installation help
VDT 14
Testing
• VDT team is building test suite• Interaction with LCG testing group• Working with NMI* to leverage:
– NMI test suite• Stress testing• Application testing (CMS pipeline)
– NMI test infrastructure
* NMI = NSF Middleware Initiative– http://www.nsf-middleware.org
VDT 15
Support
• Send us questions or problems– We will solve them if we can– We will interact with the developers, if
necessary
VDT 16
Interaction with EDG
• EDG gets Globus and Condor-G RPMs from VDT
• We do what we can to solve problems and get changes to Globus and Condor
• We want to make a great package for you
VDT 17
What exactly is in VDT?
• VDT 1.1.8:– Globus 2.2.4 + advisories + patches– Condor & Condor-G 6.5.1– Chimera/Pegasus– RLS– GLUE Schema– CA Certificates– Fault Tolerant Shell– EDG’s Make Gridmap– EDG’s CRL Update– ClassAds– Netlogger
VDT 18
Chimera Virtual Data System
• Much scientific data is not obtained from measurements but rather derived from other data by the application of computational procedures
• Chimera catalog can be used by application environments to describe a set of application programs ("transformations"), and then track all the data files produced by executing those applications ("derivations").
• Chimera contains the mechanism to locate the "recipe" to produce a given logical file, in the form of an abstract program execution graph. These abstract graphs are then turned into and executable DAG for the Condor-G DAGMan meta-scheduler by the bundled Pegasus planner.
• Enables on-demand execution of computation schedules constructed from database queries.
VDT 19
NetLogger
• “Networked Application Logger”• API w/ calls you add to existing
source code to generate time-stamped monitoring events (sent to a file, network server, syslogd, or RAM)
• Visualization Tools• Storage and Retrieval Tools
– Store all events into a database
VDT 20
Fault Tolerant Shell (FTSH)
• The Grid is a hard environment.• FTSH
– The ease of scripting with very precise error semantics.
– Exception-like structure allows scripts to be both succinct and safe.
– A focus on timed repetition simplifies the most common form of recovery in a distributed system.
– A carefully-vetted set of language features limits the "surprises" that haunt system programmers.
VDT 21
Simple Bourne script…
#!/bin/sh
cd /work/foo
rm –rf data
cp -r /fresh/data .
What if ‘/work/foo’ is unavailable??
VDT 22
Getting Grid Ready…#!/bin/sh for attempt in 1 2 3
cd /work/foo if [ ! $? ] then
echo "cd failed, trying again..." sleep 5
else break
fi done
if [ ! $? ] then
echo "couldn't cd, giving up..." return 1
fi
VDT 23
Or with FTSH
#!/usr/bin/ftsh
try 5 times
cd /work/foo
rm -rf bar
cp -r /fresh/data .
end
VDT 24
Or with FTSH
#!/usr/bin/ftsh
try for 3 days or 100 times
cd /work/foo
rm -rf bar
cp -r /fresh/data .
end
VDT 25
Or with FTSH
#!/usr/bin/ftsh
try for 3 days every 1 hour
cd /work/foo
rm -rf bar
cp -r /fresh/data .
end
VDT 26
Or with FTSH
#!/usr/bin/ftsh
try for 3 days every 1 hour
cd /work/foo
rm -rf bar
cp -r /fresh/data .
end
VDT 27
Or with FTSH
#!/usr/bin/ftsh
try for 3 days every 1 hour
cd /work/foo
rm -rf bar
cp -r /fresh/data .
end
VDT 28
Or with FTSH
hosts="mirror1.wisc.edu mirror2.wisc.edu mirror3.wisc.edu"
forany h in ${hosts} echo "Attempting host ${host}" wget http://${h}/some-file
end
echo "Got file from ${h}"
VDT 29
FTSH
• All the usual constructs– Redirection, loops, conditionals, functions,
expressions, nesting, …• And more
– Logging– Timeouts– Process Cancellation– Complete parsing at startup– File cleanup
• Used on Linux, Solaris, Irix, Cygwin, …• Simplify your life!
VDT 30
VDT’s Future
• Additional Software– MyProxy, Java ClassAds
• Access to new versions– Globus 3.0
• Extra VDT to help early adopters• Condor-G will submit to GT2 or GT3
• Helping You– What can we do to make life easier
for you?
VDT 31
Where do you learn more?
• http://www.griphyn.org/vdt• Support:
– [email protected]– Alain Roy: [email protected]– Miron Livny: [email protected]