Upload
ryval-haqul
View
220
Download
0
Embed Size (px)
Citation preview
8/9/2019 FileCatalog and Dd
1/12
8/9/2019 FileCatalog and Dd
2/12
Jerome LAURET, Coll 2
Introduction ...Introduction ...
T!e "eo"le #T!e "eo"le # $iita Soldato%, Adam &isiel, m'sel(, )$iita Soldato%, Adam &isiel, m'sel(, )
*!' do +e need a FileCatalog *!' do +e need a FileCatalog
$umber o( (iles in STAR is - 2 M .+ill get +orst, (ar +orst )/$umber o( (iles in STAR is - 2 M .+ill get +orst, (ar +orst )/
n(ormation struture om"le n(ormation struture om"le production, libraryproduction, libraryfiletype, size, geometryfiletype, size, geometrycollision, magnetic field, trigger setup namecollision, magnetic field, trigger setup name
but +e .are su""osed to/ ee" in(ormation about triggers and but +e .are su""osed to/ ee" in(ormation about triggers andounters, (inding a data4set re5uires strong Cataloguingounters, (inding a data4set re5uires strong Cataloguing
A6
7ne eisting om"lete user A6 .+ritten in "erl/, some C7ne eisting om"lete user A6 .+ritten in "erl/, some Ca ommand line inter(aea ommand line inter(ae
% get_file_list.pl% get_file_list.pl
8/9/2019 FileCatalog and Dd
3/12
Jerome LAURET, Coll 3
How do I use it ??How do I use it ??
8etting a 5ui !el" reminder8etting a 5ui !el" reminder% get_file_list.pl% get_file_list.pl
... bla bla bla ... some help that is ...
allavailable bbc collision configuration createtimedatetaken eemc emc events extension filecomment filenamefileseq filetype fpd ftpc fulld fulls gencomment generatorgenparams genversion geometryinserttime lgnm lgpthlibrary limitmagscale magvalue md5sum node noroundnouniqueowner pathpersistent pmd prodcomment productionprotection rich runcomments runnumber runtype sanitysimcomment simulationsitesitecmt siteloc size ssdstartrecordstoragestream svt tof tpc trgcounttrgdefinition trgname trgsetupnametrgversion trgword
9oumentation is a%ailable at9oumentation is a%ailable at/STAR/comp/sofi/FileCatalog//STAR/comp/sofi/FileCatalog/
8/9/2019 FileCatalog and Dd
4/12
Jerome LAURET, Coll 4
SyntaxSyntax
8eneral s'nta . :;: indiates o"tional list :
8/9/2019 FileCatalog and Dd
5/12
Jerome LAURET, Coll 5
6ossible 7"erators6ossible 7"erators
B $ot greater t!an
B Lesser t!an
D $ot less t!an
D 8reater t!an
BD $ot e5ual to e5ual to
- $ot ontaining .ie do not mat!/ strings
- Containing .ie a""roimatel' mat!ing/ strings
G n rangeG 7utside t!e range
H Modulo integer
HH $ot Modulo integer
8/9/2019 FileCatalog and Dd
6/12
Jerome LAURET, Coll 6
Welcome to the World ofWelcome to the World of
replica Catalogs.replica Catalogs. $umber o( (iles in STAR - 2 M$umber o( (iles in STAR - 2 M
T!at@s a lie Total 3 M +it! re"lias # File !a%e more t!an one loationT!at@s a lie Total 3 M +it! re"lias # File !a%e more t!an one loation
sitesite Ie a+are o( siteI$L, siteLILIe a+are o( siteI$L, siteLIL
nodenode loal!ost b' de(ault loal!ost b' de(ault
storagestorage $FS, loal, K6SS $FS, loal, K6SS"at!"at! itsel( +it!in a storageitsel( +it!in a storage
unonstraint, "at! and (ilename are $7T uni5ue e' "airsunonstraint, "at! and (ilename are $7T uni5ue e' "airs.use "distinctto ensure it onefileensures one instane o( a (ile/
$umber o( (iles on entralied storage # 1NOP$umber o( (iles on entralied storage # 1NOP$FS, dis %isible (rom an'+!ere in t!e (ailit' ."at! - ?star?dataQQ/$FS, dis %isible (rom an'+!ere in t!e (ailit' ."at! - ?star?dataQQ/
$umber o( (iles on loal dis # 131PP$umber o( (iles on loal dis # 131PP
loal dis are %isible onl' (rom a uni5ue nodeloal dis are %isible onl' (rom a uni5ue node
8/9/2019 FileCatalog and Dd
7/12
Jerome LAURET, Coll
Database layoutDatabase layout
Loations ? Re"lias
!"SS
#$S
lo%al
Fileypes
!roductionConditions
"un!arams
FileData
File
#ocations
Storageypes
StorageSites
Meta 9ata
Site, no&e, storage an& 'at( )orms t(e uni*ue +e )orFileLoations/tmp/bla.rootcannot be uni$ueBNL somenode.domain NFS /tmp/bla.root IS
%.&
$1
$1
1$
1$
1$
8/9/2019 FileCatalog and Dd
8/12
Jerome LAURET, Coll -
T'"ial Eam"lesT'"ial Eam"les
Ko+ to loate (iles +it!in a s"ei(i trigger setu" Ko+ to loate (iles +it!in a s"ei(i trigger setu" % get_file_list.pl keys pathfilename condtrgsetupname)/011ombined
+ill lead to a long .100 reords/ list o( "ossible (iles +it! "at!+ill lead to a long .100 reords/ list o( "ossible (iles +it! "at!
% get_file_list.pl keys storage condtrgsetupname)/011ombined
t!is +ill gi%e 'ou all "ossible storage t'"e (or t!e trigger setu" namet!is +ill gi%e 'ou all "ossible storage t'"e (or t!e trigger setu" nameU6CCombinedU6CCombined
n general, (or listing all "ossible %alues (or a e'+ord, usen general, (or listing all "ossible %alues (or a e'+ord, use
H get(ilelist"l 4e's e'+ord distint ;4alls
8/9/2019 FileCatalog and Dd
9/12
Jerome LAURET, Coll .
T'"ial Eam"lesT'"ial Eam"les
Iut but ) al+a's get onl' 100 reordsIut but ) al+a's get onl' 100 reords
T!at@s normal, it is t!e de(ault Use limit to !ange t!e number o( reords,T!at@s normal, it is t!e de(ault Use limit to !ange t!e number o( reords,
(ull list +it! limit 0(ull list +it! limit 0
A (e+ !and' 5uerriesA (e+ !and' 5uerries
no+ a simulation (ile name, !o+ do get t!e geometr' on(iguration no+ a simulation (ile name, !o+ do get t!e geometr' on(iguration % get_file_list.pl "keys geometry "cond% get_file_list.pl "keys geometry "condfilename)rcf2$83_2&_322evts.geant.root "distinctfilename)rcf2$83_2&_322evts.geant.root "distinct
9ear&22$9ear&22$
*!i! "rodution and geometr' *!i! "rodution and geometr' % get_file_list.pl "keys productiongeometry "cond% get_file_list.pl "keys productiongeometry "condfilename)rcf2$83_2&_322evts.geant.root "distinctfilename)rcf2$83_2&_322evts.geant.root "distinct
02$glyear&22$02$glyear&22$
02$gkyear&22$02$gkyear&22$
02&gbyear&22$02&gbyear&22$
8/9/2019 FileCatalog and Dd
10/12
Jerome LAURET, Coll /0
Aggregate 7"erationAggregate 7"eration
Can also do 5ueries leading to summar'Can also do 5ueries leading to summar'
in(ormationin(ormation
% get_file_list.pl keys% get_file_list.pl keys
:sum;sanity3524534&5&&=&8>$$&82production1entral$&22
>354$$>>3582&&4&>=53==&production1entral>22>354$$>>3582&&4&>=53==&production1entral>22
882824>$2$$$>&&8$>$2=$0roduction6in?ias882824>$2$$$>&&8$>$2=$0roduction6in?ias
8/9/2019 FileCatalog and Dd
11/12
Jerome LAURET, Coll //
7ne more one"t (uture7ne more one"t (uture
T!e e'+ord sanit' is used (or t+o aseT!e e'+ord sanit' is used (or t+o aseT!e (ile is orru"tedT!e (ile is orru"ted.R77T 7 +ill ras! 'our a""liation/.R77T 7 +ill ras! 'our a""liation/T!e (ile is $7T good (or 6!'sisT!e (ile is $7T good (or 6!'sis
ou MUST use sanit'1 to get t!e good (iles
Future .not 'et a%ailable/Future .not 'et a%ailable/
% get_file_list.pl keys pathfilename cond% get_file_list.pl keys pathfilename condtrgname)pp?@A$fastBBpp+07wfastsanity)$trgname)pp?@A$fastBBpp+07wfastsanity)$
alread' :in "lae=, onl' need to (ill t!e database onsistentl'alread' :in "lae=, onl' need to (ill t!e database onsistentl'.not done t!is 'ear/.not done t!is 'ear/
% get_file_list.pl "keys pathfilename "cond% get_file_list.pl "keys pathfilename "condtpcCD)$ftpcCD)$sanity)$'tpcCD)$ftpcCD)$sanity)$'
$ot im"lemented, +e "lan to add a detetor readiness (lag$ot im"lemented, +e "lan to add a detetor readiness (lag
8/9/2019 FileCatalog and Dd
12/12
Jerome LAURET, Coll /2
9istributed dis 9istributed dis
S!all sort t!is manuall' S!all sort t!is manuall' ou an al+a's as (orou an al+a's as (or% get_file_list.pl "cond nodepathfilename "cond% get_file_list.pl "cond nodepathfilename "condstorage)localsanity)$'storage)localsanity)$'and dis"at! b' !and ut +!' and dis"at! b' !and ut +!'
T!e S!edulerT!e S!eduler
9oes t!is (or 'ou .eam"les in net tal/ #9oes t!is (or 'ou .eam"les in net tal/ # fileEist,yntaxfileEist,yntax,,prefer,torageprefer,torage
T!ere is $7 need to useT!ere is $7 need to use "distinct"distinctoror "onefile"onefile
$otes$otes
es, "lease, use t!e sanit' (lag )es, "lease, use t!e sanit' (lag )
Use t!e S!eduler .it is a e' om"onent o( our 8rid a""roa!/Use t!e S!eduler .it is a e' om"onent o( our 8rid a""roa!/An' S!eduler /FE)Gcatalogstar.bnl.govH...an .and s!ould/ be!eed (rom t!e ommand line using get(ilelist"l ( it does not +or(rom t!e ommand line, it is $7T a S!eduler "roblem