EGRIS-1

www.eu-eela.org

E-infrastructure shared between Europe and Latin America

AMGA Metadata Services: examples and usage scenarios

Tony Calanducci

INFN Catania

Itacuruca, 04-15 December 2006

EGRIS-1

E-infrastructure shared between Europe and Latin America

Outline

• AMGA Basics– Metadata concepts and AMGA terminology– Collection and Schema management– Queries– Command Line Tools and API

• Use AMGA to exchange data among running grid jobs

• Secure access to metadata

EGRIS-1

E-infrastructure shared between Europe and Latin America

Metadata concepts and terminology

– Entries – Entities/Objects which we are attaching metadata to– Metadata - List of attributes (including their values) associated

with entries– Schema – A set of attributes– Attribute – key/value pair

Type – The type (int, float, string,…) Name/Key – The name of the attribute Value - Value of an entry's attribute

– Collection – A set of entries associated with a schema (AMGA Collection are hierarchical organized)

– Analogy to the RDBMS world: think of collections as tables, attributes as columns, entries as rows

EGRIS-1

E-infrastructure shared between Europe and Latin America

Example: Movie Trailers

• Movie trailers files saved on Grid Storage Elements and registered into a LFC File Catalogue

• We have a LFN (Logical File Name) per movie files• We want to add metadata to describe movie content.

Possible schema:– Title -- varchar– Rutime -- int– Cast -- varchar– LFN -- varchar

• AMGA will be the repository of the movies’ metadata

EGRIS-1

E-infrastructure shared between Europe and Latin America

gMOD: grid Movie On Demand

• gMOD provides a Video-On-Demand service• User chooses among a list of video and the chosen one

is streamed in real time to the video client of the user’s workstation

• For each movie a lot of details (Title, Runtime, Country, Release Date, Genre, Director, Case, Plot Outline) are stored and users can search a particular movie querying on one or more attributes

• Two kind of users can interact with gMOD: TrailersManagers that can administer the db of movies (uploading new ones and attaching metadata to them); GILDA VO users (guest) can browse, search and choose a movie to be streamed.

EGRIS-1

E-infrastructure shared between Europe and Latin America

gMOD under the hood

• Built on top of gLite services:• Storage Elements, sited in different place, physically

contain the movie files• LFC, the File Catalogue, keeps track in which Storage

Element a particular movie is located • AMGA is the repository of the detailed information for

each movie, and makes possible queries on them• The Virtual Organization Membership Service (VOMS)

is used to assign the right role to the different users • The Workload Management System (WMS) is

responsible to retrieve the chosen movie from the right Storage Element and stream it over the network down to the user’s desktop or laptop

EGRIS-1

E-infrastructure shared between Europe and Latin America

gMOD interactions

VOMS

LFC FileCatalogue

MetadataCatalogue

WN WN

WN

CE

Storage Elements

User

Genius Portal

Workload Management System

get RoleAMGA

EGRIS-1

E-infrastructure shared between Europe and Latin America

gMOD screenshot

gMOD is accesible through the Genius Portal (https://glite-tutor.ct.infn.it)

EGRIS-1

E-infrastructure shared between Europe and Latin America

Accessing AMGA services from UI/WNs

• Command Line Tools:– mdclient (interactive, like mysql & psql CLI)– mdcli (run one single command, useful into scripts)

• APIs:– C++– Java– Python/Perl/PHP

• Client configuration file: – /opt/glite/etc/mdclient.config (default for all UI users)– $HOME/.mdclient.config (overrides the default one)– My suggestion, copy the default into $HOME/.mdclient.config and

customize it– Useful options to set up:

Host = egris196.eela.ufrj.br Login = NULL (or your AMGA username we’ve created: itacurucaXX) UseSSL = require AuthenticateWithCertificate = 1 UseGridProxy = 1

EGRIS-1

E-infrastructure shared between Europe and Latin America

Accessing AMGA services from UI/WNs

• Be sure you have a valid proxy– with VOMS extensions if you set up Login = NULL

(authenticated as the gilda generic user….all VO users will see your entries)

– if you set up Login = yourusername the VOMS extensions will not take into account (so do not care about them)

• Start the AMGA client with– $ mdclient

• Try– whoami– pwd– help– grp_member

EGRIS-1

E-infrastructure shared between Europe and Latin America

AMGA Basic Commands(I)

• Entries/Collection listing– dir– ls (ls -l)

• Entries/Collection permissions– acl_show entry | collection– stat entry | collection

• Collection and schema creation :– createdir collection_pathname

createdir /egris/yoursurname– cd collection_pathname) --- change the current collection

cd /egris/yoursurname– createdir trailers– cd trailers– addattr dir attr_name attr_type

addattr . Title varchar addattr . Runtime int addattr . Cast varchar addattr . LFC varchar

EGRIS-1

E-infrastructure shared between Europe and Latin America

AMGA Basic Commands (II)

• Inspecting collection’s schema:– listrattr fullpath_collectioname

(listrattr /egris/yoursurname/trailers

• Sequence managements:– they are used to create autoincremental IDs– sequence_create seq_name dir [increment] [start value]

sequence_create seq1 /egris/yoursurname/trailers

– sequence_next seq_name sequence_next /egris/yoursurname/trailers/seq1

• Entries creations:– addentries entry_name (attribute_name value)+

cd /egris/yoursurname/trailers addentry 1 Title 'Matrix' Runtime 110 Cast 'Keanu Reeves‘

LFN 'lfn:/grid/gilda/tony/matrix.mov‘ addentry 2 Title 'Notting Hill' Runtime 99 Cast 'Julia Roberts, Hugh Grant'

LFN 'lfn:/grid/gilda/tony/notting.mov‘ addentry 3 Title 'Anger Management' Runtime 120 Cast 'Adam Sandlan,

Jack Nicholson' LFN 'lfn:/grid/gilda/scardaci/anger.mov'

EGRIS-1

E-infrastructure shared between Europe and Latin America

Other basic useful commands

• Set/edit attributes values:– setattr entry (attribute value)+

• Get attributes values:– getattr pattern (attribute)+

getattr /egris/yoursurname/trailers/ Title LFN

• Removing attributes– removeattr dir attr_name

• Removing entries– rm pattern

EGRIS-1

E-infrastructure shared between Europe and Latin America

Making queries

• AMGA provides a SQL-like selectattr statement:– selectattr attrs... condition

where:• attrs is a space-separated list of attributes with the following

format: collection_name:attribute_name (ex.: /egris/yoursurname/trailers:Title)

• condition rapresents one or more condition to select entries. Allowed operators are comparison operator (<,>,=), logical operator (and, or), various operator (like, limit), etc..There is support also for aggregator operator (count) and order clause.

• ex: – selectattr /egris/yoursurname/trailers:Title .:LFN 'like(Title,

"Anger%")' – selectattr /egris/yoursurname/trailers:Title .:LFN .:Cast 'like(Cast,

"%ndle%") and Runtime > 80'

EGRIS-1

E-infrastructure shared between Europe and Latin America

Updating attributes on condition

• selectattr and setattr can be used to update entries attributes’ values

• updateattr allows to do multiple updating on condition and ATOMICALLY– updateattr pattern attr expression [attr expression]... condition– ex:

getattr /egris/yoursurname/trailers/ Runtime >> 2>> 99>> 1>> 111>> 3>> 121

updateattr /egris/yoursurname/trailers/ Runtime Runtime+1 'Runtime > 100‘ getattr /egris/yoursurname/trailers/ Runtime

>> 2>> 99>> 1>> 112>> 3>> 122

EGRIS-1

E-infrastructure shared between Europe and Latin America

Access AMGA from WNs with mdcli

• All commands available through mdclient can be issued with mdcli <command>. This will:– connect to the AMGA server defined into mdclient.config– run the command– exit to the shell

• Very useful in shell script (ex. the one running in the WN during job execution)

• Ex:– ID=`mdcli sequence_next /egris/yoursurname/seq1`– mdcli addentry /egris/yoursurname/${ID} LFN \'${LFN}\'

Analyzed 0

– LFN=`mdcli selectattr $AMGA_HOME:LFN \'$AMGA_HOME:FILE = ${ID}\'`

• N.B.: If you need to use single quote, escape them with \

EGRIS-1

E-infrastructure shared between Europe and Latin America

Use AMGA to exchange data among running jobs

• Suppose we have two sets of jobs: – Producers: they generate a file, store on a SE, register it onto

the LFC File Catalogue assigning a LFN– Consumers: they will take a LFN, download the file and

elaborate it

• AMGA can be used to share the information generated by the Producers, it could act as a “bag-of-LFNs” (bag-of-task model) from which Consumers can fetch file for further elaboration

EGRIS-1

E-infrastructure shared between Europe and Latin America

Producers-Consumers example

• Set up of the collection containing the “tasks”– createdir /egris/youraccount/tasks– cd /egris/youraccount/tasks– addattr . LFN varchar(255)– addattr . Analyzed int– addattr . MagicNumber int– listattr .>> LFN>> varchar(255)>> Analyzed>> int>> MagicNumber>> int– sequence_create seq1 /egris/youraccount/tasks

EGRIS-1

E-infrastructure shared between Europe and Latin America

Producers-Consumers example

• Producers script excerpts:– #Produce your data and save it on the SE and reg on LFC– LFN=‘lfn:/grid/gilda/tony/bomdia.txt’– ID=`mdcli sequence_next /egris/yourname/tasks/seq1`– mdcli addentry /egris/yourname/tasks/${ID} LFN \'${LFN}\' Analyzed 0

• Consumers script excerpts:– AMGA_HOME=/egris/yoursurname/tasks – ID=`mdcli selectattr $AMGA_HOME:FILE \'$AMGA_HOME:Analyzed = 0\' |

head -1`– if [ $ID ]; then; echo “Task to be processed found“; else; exit 1; fi– MAGIC=`echo $RANDOM`– mdcli updateattr $AMGA_HOME/$ID Analyzed 1 MagicNumber

$MAGIC \'Analyzed = 0\'– MAGIC2=`mdcli selectattr

$AMGA_HOME:MagicNumber \'$AMGA_HOME:FILE = ${ID}\'`– if [ $MAGIC=$MAGIC2 ]; then

LFN=`mdcli selectattr $AMGA_HOME:LFN \'$AMGA_HOME:FILE = ${ID}\'`– else; echo "Entry already processed by someone else“; exit 1; fi– lcg-cp -v --vo gilda $LFN file:$PWD/temp.dat– #Consume your data

EGRIS-1

E-infrastructure shared between Europe and Latin America

Demo

• Thanks to Fabio for the creations of the scripts!!!

EGRIS-1

E-infrastructure shared between Europe and Latin America

Secure your metadata

• AMGA allows to restrict access to a collection (or even to specific entries) using ACLs. This can be done thanks to the commands provided to manage groups of users.

• Here there are some useful commands:– whoami -- Print the name of the current user– user_listcred user -- Lists all possible credentials of a user. Only root

can inspect other's credentials.– grp_member -- Lists the groups a user belongs to– acl_show colletionpath -- Shows all access controls for a directory– grp_create groupname -- Creates a new group. The group created will

have the following name owner:groupName– grp_adduser groupname user -- Adds a user to a group– grp_show groupname -- Lists the members of a group– acl_add directory group rights -- Adds a new access control to a

directory granting a group certain rights– acl_remove directory group -- Removes an access control for a group

from a directory

EGRIS-1

E-infrastructure shared between Europe and Latin America

References

• AMGA Project Homepage– http://amga.web.cern.ch/amga/

• AMGA 1.2.7 User Manual– http://amga.web.cern.ch/amga/downloads/amga-manual_1_2_7.

pdf

• Exercise documentation from ISSGC’06:– http://www.dma.unina.it/~murli/ISSGC06/glite/public_html/amga.

html– http://www.dma.unina.it/~murli/ISSGC06/glite/public_html/summ

aryExercise.htm

• AMGA GILDA Wiki pages:– https://grid.ct.infn.it/twiki/bin/view/GILDA/AMGAHandsOn– https://grid.ct.infn.it/twiki/bin/view/GILDA/AMGAAdv