EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI proposal for a Storage Accounting Record standard

Jon Kerr NilsenDept. of Physics, Univ. of Oslo

On behalf of the EMI StAR team

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• EMI StAR team– Paul Millar– Ralph Müller-Pfefferkorn– Zsolt Molnár– Riccardo Zappi– Jon Kerr Nilsen

• Henrik Thostrup Jensen (NDGF)

Who?

Jon Kerr Nilsen 2

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Why storage accounting?• Storage accounting vs. job accounting• Processing model• Storage accounting work in EMI and OGF• StAR proposal• Prototype and first feedback• Way forward

Outline

Jon Kerr Nilsen 3

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Why storage accounting?

• Storage resources are shared between organizations – more than one owner

• Developing a storage infrastructure– We need to know how much

storage space is used, by which group/user, on which storage media

– To know where to put the money when increasing the storage space

– To know who to ask for the money to increase the storage space

• Basis for billing– Storage is expensive – Some non-academic resource

owners may not like to give it away for free

Jon Kerr Nilsen 4

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Accounting

• Accounting is the recording and summarizing of the consumption of a resource by an individual user or a group of users

• Typically used to find out who uses how much of a set of resources

• Typically not used to find out how individual resource components are used

Jon Kerr Nilsen 5

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Storage vs. job accounting

Job accounting• Job accounting is the recording

and summarizing of the consumption of computing resources

• Recording wall-time, cpu-time, etc. – strictly increasing values.

• When a job finishes it stays finished

• Typically measured per job – information readily available from local batch system

• Standardized through UR1.0

Storage accounting• Storage accounting is the recording and

summarizing of the consumption of a storage resource

• Storage usage varies in time – recorded usage is only relevant in a given time frame

• Data is rarely in finished state and can live for a long time

• Typically recorded through (in)frequent usage snapshots – high fluctuation rate

• No standardized way to record storage usage in a distributed environment

Jon Kerr Nilsen 6

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Storage vs. job summary

Jon Kerr Nilsen 7

• Storage records must be treated different than compute records wrt. aggregation

• Storage records need special treatment– Only valid between measurement time and

expiration time– Can be invalidated if newer record is

registered for same resource– Nontrivial processing model

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Processing model

• Records can overlap in time – cannot simply sum up to identify total consumption

• Must ask for a certain point in time and sum up the most recent, still-valid records

• To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration

Jon Kerr Nilsen 8

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Processing model

• Records can overlap in time – cannot simply sum up to identify total consumption

• Must ask for a certain point in time and sum up the most recent, still-valid records

• To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration

Jon Kerr Nilsen 9

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Processing model

• Records can overlap in time – cannot simply sum up to identify total consumption

• Must ask for a certain point in time and sum up the most recent, still-valid records

• To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration

Jon Kerr Nilsen 10

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Processing model

• Records can overlap in time – cannot simply sum up to identify total consumption

• Must ask for a certain point in time and sum up the most recent, still-valid records

• To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration

Jon Kerr Nilsen 11

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Processing model

• Records can overlap in time – cannot simply sum up to identify total consumption

• Must ask for a certain point in time and sum up the most recent, still-valid records

• To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration

Jon Kerr Nilsen 12

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Processing model

• Records can overlap in time – cannot simply sum up to identify total consumption

• Must ask for a certain point in time and sum up the most recent, still-valid records

• To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration

Jon Kerr Nilsen 13

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Processing model

• Records can overlap in time – cannot simply sum up to identify total consumption

• Must ask for a certain point in time and sum up the most recent, still-valid records

• To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration

Jon Kerr Nilsen 14

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• From EMI Description of Work:– EMI will address known UR issues and extend

accounting records to include VO-aware storage usage accounting

– The refined and extended standard UR format, service interface and data transport protocol will be implemented in gLite, ARC, dCache and UNICORE

– EMI will thus contribute to the definition of UR 2.0 of OGF

Storage accounting in EMI

Jon Kerr Nilsen 15

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• “Next Generation Usage Records” OGF session at OGF31– Discussed the way forward for UR– A properly defined storage record and an improved

compute record are good steps towards UR2.0– Lessons learned in storage records can be ported

back into compute records (e.g. long-lived services)– EMI will work actively to have standard ready before

end of EMI• Being a highly active participant, EMI will co-chair

the OGF UR-WG

Storage accounting in OGF

Jon Kerr Nilsen 16

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• EMI is finalizing a proposal for a storage accounting record (StAR)

• Contacted and received input from organizations and potential user groups– NDGF, OGF, EGI, OSG, INFN, CMS, ATLAS, …

• Current draft: StAR-EMI-tech-doc-v7.pdf• Final version to be proposed to OGF as a

standard• Will be implemented by EMI storage providers

StAR

Jon Kerr Nilsen 17

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• To be able to arrive to a useful record in record time we need to limit the scope

• Many interesting storage-related features to record– I/O (like in Amazon S3)– Metadata access (stat,ls)– Available space– File name– ...

• Scope of StAR is limited to consumption of storage space

What StAR is not

Jon Kerr Nilsen 18

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Resource– Fields describing the system the resource was consumed on– StorageSystem, StorageShare

• Resource consumption– How much of the described resource has been used– ResourceCapacityUsed, LogicalCapacityUsed

• Consumption details– Fields describing what is consumed– StorageMedia, StorageClass, DirectoryPath, …

• Identity– Describes the person or group accountable for the consumption– SubjectIdentity block (UserIdentity, Group, GroupAttribute, …)

• Record identity and duration

StAR structure

Jon Kerr Nilsen 19

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Local example

Jon Kerr Nilsen 20

<sr:StorageUsageRecord xmlns:sr="http://eu-emi.eu/namespaces/2011/02/storagerecord"> <sr:RecordIdentity sr:createTime="2010-11-09T09:06:52Z" sr:recordId="host.example.org/sr/87912469269276"/> <sr:StorageSystem>host.example.org</sr:StorageSystem> <sr:SubjectIdentity> <sr:LocalUser>johndoe</sr:LocalUser> </sr:SubjectIdentity> <sr:StorageMedia>ssd</sr:StorageMedia> <sr:FileCount>42</sr:FileCount> <sr:MeasureTime>2010-10-11T09:31:40Z</sr:MeasureTime> <sr:ValidDuration>PT3600S</sr:ValidDuration> <sr:ResourceCapacityUsed>913617</sr:ResourceCapacityUsed></sr:StorageUsageRecord>

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

<sr:StorageUsageRecord xmlns:sr="http://eu-emi.eu/namespaces/2011/02/storagerecord"> <sr:RecordIdentity sr:createTime="2010-11-09T09:06:52Z" sr:recordId="host.example.org/sr/87912469269276"/> <sr:StorageSystem>host.example.org</sr:StorageSystem> <sr:SubjectIdentity> <sr:Group>binarydataproject.example.org</sr:Group> <sr:GroupAttribute sr:attributeType="subgroup">ukusers </sr:GroupAttribute> </sr:SubjectIdentity> <sr:StorageMedia>tape</sr:StorageMedia> <sr:FileCount>42</sr:FileCount> <sr:MeasureTime>2010-10-11T09:31:40Z</sr:MeasureTime> <sr:ValidDuration>PT3600S</sr:ValidDuration> <sr:ResourceCapacityUsed>913617</sr:ResourceCapacityUsed></sr:StorageUsageRecord>

Grid example

Jon Kerr Nilsen 21

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• NDGF already has working prototype– Using SGAS accounting service and data from NDGF

dCache• Records inserted in SGAS, parsed and put into

PostgreSQL DB• Querying data model with measure points and valid

duration doable for anyone comfortable with putting joins in queries

• Hackish record generation (simple SQL into dCache DB)

• Still limited graphing possibilities

Prototype implemented

Jon Kerr Nilsen 22

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Prototype implemented

Jon Kerr Nilsen 24

3/30/1

1

3/31/1

1

4/1/1

1

4/2/1

1

4/3/1

1

4/4/1

1

4/5/1

1

4/6/1

1

4/7/1

1

4/8/1

10

500

1000

1500

2000

2500

3000

aliceatlas-dkatlas-noatlas-prodatlas-userbehrmanndteamops

Usa

ge (

TB

)

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Information available from records has limitations (we knew that)

• Good for measuring actual usage• Good for measuring logical usage/VO• Logical usage/site/VO is more tricky• Measuring and aggregating logical usage is

not straight-forward• Free space could be useful as well but this

is tricky for same reasons as logical usage

First experiences from NDGF

Jon Kerr Nilsen 25

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• The NDGF dCache system is composed of seven sites, each storing a lot of data

• NDGF replicates all data which has been accessed within the last 30 days to two sites

• NDGF needs usage per VO and per site

Local usage issue

Jon Kerr Nilsen 26

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Local usage issue

• If a file is stored on site A and site B for a VO C, then what exactly is reported as logical usage for any of the sites for that VO?

• Cannot say from record if file* is a replica or a logical file

• May need hierarchy in the record to get clear answer– More complex, less intuitive and

more error prone record– Different storage systems may have

different hierarchies – generic data model will be REALLY complicated

• We probably don’t want to go there

Jon Kerr Nilsen 27

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Definition of StAR is a first step to establish common storage accounting record

• A step towards a common usage record (UR2.0)• Next jump (two steps in one go)

– Propose StAR to OGF to start standardization process• Will take time• Will include further discussions• May lead to changes in StAR• EMI will take active part in this process• Draft has been made available through UR-WG mailing list• Will be made available in SourceForge

– Implement StAR record into EMI storage middleware• EMI ends in two years• Want to gain user experience before end of EMI• Work already started outside EMI

• Comments and input more than welcome throughout the process!

Way forward

Jon Kerr Nilsen 28

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

THANK YOU!

Contact: emi-jra1-data-sar@eu-emi.euWeb page: http://twiki.cern.ch/twiki/bin/view/EMI/StorageAccounting

Current StAR draft: http://twiki.cern.ch/twiki/pub/EMI/StorageAccounting/StAR-EMI-tech-doc-v7.pdf

http://twiki.cern.ch/twiki/pub/EMI/StorageAccounting/StAR-EMI-tech-doc-v7.doc

Jon Kerr Nilsen 29