ATLAS DDM Operations - III DPM at T2’s

INFSO-RI-508833

Enabling Grids for E-sciencE

www.eu-egee.org

ATLAS DDM Operations - IIIDPM at T2’s Jiří Chudoba

ATLAS meeting, 25.9.2007, CNAF

ATLAS DDM Operations 2


INFSO-RI-508833

DPM at T2’s

• FZK cloud:– FZU

srm://golias100.farm.particle.cz/dpm/farm.particle.cz/home/atlas/ – CYF srm://dpm.cyf-kr.edu.pl/dpm/cyf-kr.edu.pl/home/atlas/

• SARA cloud:– NIKHEF: srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/ – IHEP: srm://dpm0001.m45.ihep.su/dpm/m45.ihep.su/home/atlas/ – PNPI: srm://cluster.pnpi.nw.ru/dpm/pnpi.nw.ru/home/atlas/dq2

• CNAF cloud:– Milano: srm://grid015.mi.infn.it/dpm/mi.infn.it/home/atlas/– Roma1:

srm://grid-cert-03.roma1.infn.it/dpm/roma1.infn.it/home/atlas/ – Napoli: srm://t2-dpm-01.na.infn.it/dpm/na.infn.it/home/atlas/ – LNF: srm://atlasse.lnf.infn.it/dpm/lnf.infn.it/home/atlas/



INFSO-RI-508833

DPM

• DPM documentation:– https://twiki.cern.ch/twiki/bin/view/LCG/DataManagementDocumentation

• Description of DBs on some wiki pages:http://www.gridpp.ac.uk/wiki/DPM_MySQL_database

• DBs: cns_db, dpm_db• Tables from cns_db: Cns_file_metadata,

Cns_file_replica, Cns_groupinfo, Cns_userinfo



INFSO-RI-508833

Cns_file_metadata

+---------------+---------------------+------+-----+---------+----------------+| Field | Type | Null | Key | Default | Extra |+---------------+---------------------+------+-----+---------+----------------+| rowid | bigint(20) unsigned | | PRI | NULL | auto_increment || fileid | bigint(20) unsigned | YES | MUL | NULL | || parent_fileid | bigint(20) unsigned | YES | MUL | NULL | || guid | varchar(36) | YES | MUL | NULL | || name | varchar(255) | YES | | NULL | || filemode | int(10) unsigned | YES | | NULL | || nlink | int(11) | YES | | NULL | || owner_uid | int(10) unsigned | YES | | NULL | || gid | int(10) unsigned | YES | | NULL | || filesize | bigint(20) unsigned | YES | | NULL | || atime | int(11) | YES | | NULL | || mtime | int(11) | YES | | NULL | || ctime | int(11) | YES | | NULL | || fileclass | smallint(6) | YES | | NULL | || status | char(1) | YES | | NULL | || csumtype | char(2) | YES | | NULL | || csumvalue | varchar(32) | YES | | NULL | || acl | blob | YES | | NULL | |+---------------+---------------------+------+-----+---------+----------------+



INFSO-RI-508833

Cns_file_replica

+------------+---------------------+------+-----+---------+----------------+| Field | Type | Null | Key | Default | Extra |+------------+---------------------+------+-----+---------+----------------+| rowid | bigint(20) unsigned | | PRI | NULL | auto_increment || fileid | bigint(20) unsigned | YES | MUL | NULL | || nbaccesses | bigint(20) unsigned | YES | | NULL | || atime | int(11) | YES | | NULL | || ptime | int(11) | YES | | NULL | || status | char(1) | YES | | NULL | || f_type | char(1) | YES | | NULL | || poolname | varchar(15) | YES | | NULL | || host | varchar(63) | YES | MUL | NULL | || fs | varchar(79) | YES | | NULL | || sfn | blob | YES | MUL | NULL | || ctime | int(11) | YES | | NULL | || ltime | int(11) | YES | | NULL | || r_type | char(1) | YES | | NULL | || setname | varchar(36) | YES | | NULL | |+------------+---------------------+------+-----+---------+----------------+



INFSO-RI-508833

Cns_userinfo

+----------+------------------+------+-----+---------+----------------+| Field | Type | Null | Key | Default | Extra |+----------+------------------+------+-----+---------+----------------+| rowid | int(10) unsigned | | PRI | NULL | auto_increment || userid | int(11) | YES | | NULL | || username | varchar(255) | YES | MUL | NULL | |+----------+------------------+------+-----+---------+----------------+



INFSO-RI-508833

Cns_groupinfo

+-----------+------------------+------+-----+---------+----------------+| Field | Type | Null | Key | Default | Extra |+-----------+------------------+------+-----+---------+----------------+| rowid | int(10) unsigned | | PRI | NULL | auto_increment || gid | int(11) | YES | | NULL | || groupname | varchar(255) | YES | MUL | NULL | |+-----------+------------------+------+-----+---------+----------------+



INFSO-RI-508833

ATLAS groups

Examples from DPM at golias100

select * from Cns_groupinfo where groupname like '%atlas%';+-------+------+-------------------------------+| rowid | gid | groupname |+-------+------+-------------------------------+| 1 | 1309 | atlassgm || 3 | 1308 | atlas || 7 | 1307 | atlas/Role=lcgadmin || 8 | 1311 | atlasprd || 10 | 1310 | atlas/Role=production || 16 | 116 | atlas/lcg1 || 18 | 118 | atlas/usatlas || 21 | 121 | atlas/ca || 30 | 2695 | atlas/de || 32 | 2697 | atlas/nl || 33 | 2698 | atlas/fr || 35 | 2700 | atlas/usatlas/Role=production |+-------+------+-------------------------------+

• Until this summer, we mapped all ATLAS users to gid=1307



INFSO-RI-508833

ATLAS users select owner_uid, username, sum(filesize)/1000000000 as size_GB from Cns_file_metadata, Cns_userinfo, Cns_groupinfo where

owner_uid=userid and Cns_groupinfo.gid=Cns_file_metadata.gid and Cns_groupinfo.groupname like "atlas%" group by owner_uid;+-----------+-------------------------------------------------------------------------------------------------------------+---------+| owner_uid | username | size_GB |+-----------+-------------------------------------------------------------------------------------------------------------+---------+| 101 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Guido Negri/[email protected] | 414.78 || 102 | /DC=cz/DC=cesnet-ca/O=Institute of Physics of the Academy of Sciences of the CR/CN=Jiri Chudoba | 56.85 || 104 | /C=CH/O=CERN/OU=GRID/CN=Simone Campana 7461 - ATLAS | 21.24 || 106 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Silvia Resconi/[email protected] | 95.37 || 108 | /C=CH/O=CERN/OU=GRID/CN=Jiri Kosina 6222 | 0.00 || 115 | /C=CA/O=Grid/OU=westgrid.ca/CN=Rodney Walker | 22.54 || 128 | /C=UK/O=eScience/OU=Lancaster/L=Physics/CN=james catmore | 0.00 || 132 | /O=GermanGrid/OU=LMU/CN=Gernot Krobath | 1.89 || 146 | /O=GermanGrid/OU=LMU/CN=Cedric Serfon | 0.10 || 149 | /O=GermanGrid/OU=LMU/CN=John Kennedy | 16.40 || 154 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Guido Negri 2 | 139.78 || 157 | /C=CH/O=CERN/OU=GRID/CN=Miguel De Oliveira Branco 2423 | 217.00 || 160 | /C=CA/O=Grid/OU=phys.uvic.ca/CN=Ashok Agarwal | 1.12 || 166 | /O=GermanGrid/OU=LMU/CN=Johannes Elmsheuser | 0.00 || 189 | /C=PL/O=GRID/O=INP/CN=Andrzej Olszewski | 3.10 || 191 | /C=TW/O=AS/OU=PHYS/CN=suijian zhou/[email protected] | 2.29 || 192 | /O=GermanGrid/OU=DESY/CN=Andreas Haupt | 1.11 || 195 | /C=UK/O=eScience/OU=QueenMaryLondon/L=Physics/CN=kondo gnanvo | 0.00 || 218 | /C=CH/O=CERN/OU=GRID/CN=Piotr Nyczyk 6217 - Atlas | 0.00 || 219 | /C=CH/O=CERN/OU=GRID/CN=David Cameron 0841 | 453.53 || 247 | /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=dcameron/CN=555105/CN=David Cameron | 0.00 || 248 | /C=PL/O=GRID/O=IFJ/CN=Anna Kaczmarska - Atlas | 0.00 || 250 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Davide Rebatto/[email protected] | 44.54 || 258 | /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=mlassnig/CN=663551/CN=Mario Lassnig | 4741.78 || 263 | /DC=org/DC=doegrids/OU=People/CN=Sanjay Padhi 496075 | 155.73 || 318 | /C=IT/O=INFN/OU=Personal Certificate/L=Roma 1/CN=Alessandro De Salvo/[email protected] | 2.12 || 334 | /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=caitriana nicholson | 2.14 || 358 | /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sch/CN=664224/CN=Ariel Gustavo Schwartzman | 0.03 |+-----------+-------------------------------------------------------------------------------------------------------------+---------+



INFSO-RI-508833

ATLAS production users select owner_uid, username, Cns_groupinfo.groupname, sum(filesize)/1000000000 as size_GB from Cns_file_metadata, Cns_userinfo, Cns_groupinfo where

owner_uid=userid and Cns_groupinfo.gid=Cns_file_metadata.gid and Cns_groupinfo.gid=1310 group by owner_uid;+-----------+-------------------------------------------------------------------------------------------------+-----------------------+---------+| owner_uid | username | groupname | size_GB |+-----------+-------------------------------------------------------------------------------------------------+-----------------------+---------+| 106 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Silvia Resconi/[email protected] | atlas/Role=production | 6.41 || 115 | /C=CA/O=Grid/OU=westgrid.ca/CN=Rodney Walker | atlas/Role=production | 3.56 || 154 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Guido Negri 2 | atlas/Role=production | 14.54 || 250 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Davide Rebatto/[email protected] | atlas/Role=production | 14.85 || 263 | /DC=org/DC=doegrids/OU=People/CN=Sanjay Padhi 496075 | atlas/Role=production | 71.11 |+-----------+-------------------------------------------------------------------------------------------------+-----------------------+---------+

select owner_uid, Cns_groupinfo.groupname, sum(filesize)/1000000000 as size_GB from Cns_file_metadata, Cns_userinfo, Cns_groupinfo where owner_uid=userid and Cns_groupinfo.gid=Cns_file_metadata.gid and Cns_groupinfo.gid=1310 group by owner_uid;

+-----------+-----------------------+---------+| owner_uid | groupname | size_GB |+-----------+-----------------------+---------+| 106 | atlas/Role=production | 6.41 || 115 | atlas/Role=production | 3.56 || 154 | atlas/Role=production | 14.54 || 250 | atlas/Role=production | 14.85 || 263 | atlas/Role=production | 71.11 |+-----------+-----------------------+---------+



INFSO-RI-508833

How old are ATLAS files

select date_format(from_unixtime(ctime),'%Y-%m') as month, count(*) as nfiles, sum(filesize)/1000000000 as size_GB from Cns_file_metadata, Cns_groupinfo where

Cns_groupinfo.gid=Cns_file_metadata.gid and Cns_groupinfo.groupname like "atlas%" group by month;

+---------+--------+---------+| month | nfiles | size_GB |+---------+--------+---------+| 2006-03 | 1920 | 127.39 || 2006-04 | 10718 | 859.46 || 2006-05 | 23333 | 1645.98 || 2006-06 | 5275 | 601.59 || 2006-07 | 506 | 35.49 || 2006-08 | 334 | 35.53 || 2006-09 | 162 | 18.39 || 2006-10 | 5414 | 665.23 || 2006-11 | 39 | 0.00 || 2006-12 | 973 | 0.00 || 2007-01 | 51 | 0.00 || 2007-02 | 159 | 0.00 || 2007-03 | 1922 | 4.94 || 2007-04 | 2336 | 72.66 || 2007-05 | 7966 | 346.48 || 2007-06 | 44197 | 1822.96 || 2007-07 | 845 | 85.89 || 2007-08 | 27705 | 1844.95 || 2007-09 | 22093 | 1020.92 |+---------+--------+---------+



INFSO-RI-508833

Access to files owned by production

select nbaccesses, count(*) from Cns_file_replica, Cns_file_metadata where Cns_file_replica.fileid=Cns_file_metadata.fileid and Cns_file_metadata.gi

d=1310 group by nbaccesses ;+------------+----------+| nbaccesses | count(*) |+------------+----------+| 1 | 978 || 2 | 825 || 3 | 183 || 4 | 30 || 5 | 30 || 6 | 11 || 7 | 5 || 8 | 1 || 10 | 2 || 11 | 1 || 12 | 2 || 13 | 1 || 14 | 2 || 16 | 1 || 19 | 2 || 22 | 3 || 30 | 1 || 31 | 3 || 32 | 5 || 33 | 1 || 67 | 1 |+------------+----------+21 rows in set (1.77 sec)

•1 access = file was only stored – no copy on T1



INFSO-RI-508833

Production files not yet replicated to T1

select md.name, date_format(from_unixtime(md.ctime),'%Y-%m-%d') as day from Cns_file_replica r, Cns_file_metadata md where r.fileid=md.fileid and md.gid=1310 and

r.nbaccesses=1 order by day asc limit 10 ;+---------------------------------+------------+| name | day |+---------------------------------+------------+| log.012663._00127.job.log.tgz.1 | 2007-08-23 || log.010685._80590.job.log.tgz.1 | 2007-08-23 || ESD.011212._03771.pool.root.4 | 2007-08-29 || EVNT.011675._00528.pool.root.2 | 2007-08-29 || log.011680._00605.job.log.tgz.2 | 2007-08-29 || log.013304._00016.job.log.tgz.3 | 2007-08-29 || RDO.013306._00003.pool.root.1 | 2007-08-29 || log.013303._00022.job.log.tgz.3 | 2007-08-29 || RDO.013304._00061.pool.root.3 | 2007-08-29 || HITS.013304._00014.pool.root.3 | 2007-08-29 |+---------------------------------+------------+



INFSO-RI-508833

Usage of files replicated by DDM



INFSO-RI-508833

AOD Usage

select count(*) from Cns_file_replica r where fileid in (select fileid from Cns_file_metadata where owner_uid=258 and name like '%AOD%');

66436 AOD files owned by DDM (Mario’s proxy)

select r.nbaccesses, count(*) from Cns_file_replica r where fileid in (select fileid from Cns_file_metadata where owner_uid=258 and name like '%AOD%') group by n

baccesses limit 15;+------------+----------+| nbaccesses | count(*) |+------------+----------+| 1 | 45411 || 2 | 11384 || 3 | 3324 || 4 | 1974 || 5 | 1629 || 6 | 755 || 7 | 762 || 8 | 402 || 9 | 202 || 10 | 188 || 11 | 135 || 12 | 70 || 13 | 56 || 14 | 34 || 15 | 24 |+------------+----------+

• 66 436 AOD files, 45411 never used via DPM, 90 files used more than 15 times



INFSO-RI-508833

dpm_db

dpm_get_filereq:+-------------+---------------------+------+-----+---------+----------------+| Field | Type | Null | Key | Default | Extra |+-------------+---------------------+------+-----+---------+----------------+| rowid | bigint(20) unsigned | | PRI | NULL | auto_increment || r_token | varchar(36) | YES | MUL | NULL | || f_ordinal | int(11) | YES | | NULL | || from_surl | blob | YES | MUL | NULL | || lifetime | int(11) | YES | | NULL | || f_type | char(1) | YES | | NULL | || s_token | varchar(36) | YES | | NULL | || flags | int(11) | YES | | NULL | || server | varchar(63) | YES | | NULL | || pfn | blob | YES | MUL | NULL | || actual_size | bigint(20) unsigned | YES | | NULL | || status | int(11) | YES | | NULL | || errstring | varchar(255) | YES | | NULL | || protocol | varchar(7) | YES | | NULL | || ret_policy | char(1) | YES | | NULL | |+-------------+---------------------+------+-----+---------+----------------+



INFSO-RI-508833

total transfers

select sum(actual_size)/1000000000 as get_GB from dpm_get_filereq ;+----------+| get_GB |+----------+| 47163.37 |+----------+

select sum(actual_size)/1000000000 as put_GB from dpm_put_filereq ;+----------+| put_GB |+----------+| 22372.59 |+----------+



INFSO-RI-508833

put

put requests:select date_format(from_unixtime(ctime),'%Y-%m') as month, count(*) from dpm_req where

r_token in (select r_token from dpm_put_filereq) group by month;+---------+----------+| month | count(*) |+---------+----------+| 2006-03 | 2549 || 2006-04 | 16604 || 2006-05 | 28188 || 2006-06 | 9132 || 2006-07 | 5342 || 2006-08 | 5116 || 2006-09 | 4370 || 2006-10 | 34282 || 2006-11 | 10414 || 2006-12 | 15870 || 2007-01 | 11820 || 2007-02 | 10638 || 2007-03 | 31518 || 2007-04 | 21938 || 2007-05 | 60971 || 2007-06 | 70478 || 2007-07 | 8267 || 2007-08 | 61377 || 2007-09 | 75111 |+---------+----------+



INFSO-RI-508833

get

Get requests:select date_format(from_unixtime(ctime),'%Y-%m') as month, count(*) from dpm_req where

r_token in (select r_token from dpm_get_filereq) group by month;+---------+----------+| month | count(*) |+---------+----------+| 2006-03 | 1039 || 2006-04 | 8063 || 2006-05 | 34846 || 2006-06 | 20797 || 2006-07 | 9131 || 2006-08 | 7142 || 2006-09 | 18359 || 2006-10 | 16221 || 2006-11 | 13251 || 2006-12 | 8350 || 2007-01 | 3842 || 2007-02 | 8873 || 2007-03 | 13731 || 2007-04 | 10503 || 2007-05 | 7373 || 2007-06 | 7551 || 2007-07 | 9663 || 2007-08 | 48762 || 2007-09 | 41102 |+---------+----------+



INFSO-RI-508833

DPM dumps

• export DPNS_HOST=t2-dpm-01.na.infn.it• export DPM_HOST=t2-dpm-01.na.infn.it• dpns-ls -lR /dpm/na.infn.it/home/atlas >

napoli.dpnsls.20070924.list



INFSO-RI-508833

CheckDpmConsistency

• https://twiki.cern.ch/twiki/bin/view/LCG/CheckDpmConsistency • Author: Lana Abadie• The DPM consistency check tool is an admin tool

which can be used for the following purposes – behaviour by default: – it allows listing files which are stored in the name server but not

physically stored; – it allows listing files which are stored physically and not declared

in the DPNS DB; – it allows listing files whose sizes stored in the name server differ

from the physical size



INFSO-RI-508833

dpns-cleanup

• By Lana Abadie• I tested it when we lost all files at 1 pool node• Uses these main calls:

– dpns_listreplicax– dpns_delreplica– dpns_getpath– dpns_unlink

• Example:./dpns-cleanup --dpns_host_name golias100.farm.particle.cz --

server se2.farm.particle.cz

started at 11:06, finished 14:22deleted 24414 files.



INFSO-RI-508833

ATLAS groups

Documents

ATLAS DDM Operations - III DPM at T2’s