1
INAF - National Institute for Astrophysics The National Institute for Astrophysics coordinates and participates in the Astronomy and Astrophysics (A&A) cluster in EGI, using extensive and large applications on the Grid. INAF is a partner of IGI (Italian Grid Initiative) and participates in its activities, with skilled personnel and with computational resources in several INAF local institutes. VisIVO tools for Large Dataset Exploration on gLite Grid The Visualisation Interface to the Virtual Observatory (VisIVO) is a suite of software tools aimed at creating 3D customized views of many type of wide used data sets. Its peculiar characteristic is that there is no limit for what concerns the size of input tables containing data to be processed, thus they are able to support very large scale datasets (tens of Terabytes and more). VisIVO is designed to create images and movies from user files. The first fundamental goal of porting VisIVO to the Grid is to have movies and images directly generated and stored on the Grid, using the internal arrays of a running code on worker nodes, and to reduce the overall time for movie production. A VisIVO library was designed from November to December 2010 and the development phase took place in the first four months of 2011. VisIVO library represent an important step in order to export the usage of the 3D visualization rendering and movie creation tools to many user communities. The user can set the environment variables to use the multithread or the multiprocess features. By default the multiprocess features are adopted. The user application may call several instances for VA_Importer, VA_Filter, VA_View and several threads or processes will be started simultaneously. VisIVO may be compiled on a gLite system 3.1. In this case the GLITE option must be given when compiling the tool. We used lcg_util APIs: an high-level gLite data management APIs. They are able to provide a single interface to deal with Storage Elements and LFC. They actually provide they same functionality available throw the lcg_util command line tools (lcg-cr, lcg-cp, lcg-rep, lcg-del, etc). We used these functions to create a local copy of the table in the local WN. User code linked with VisIVO Library User code submitted to the Grid User code Running on Wn User code produces images and movies using VisIVO Grid Catalogue Monitor the data production during the run --fformat votable /home/user/demo/vizier.xml ..... --x x --y y --z z --color -- colortable --colorscalar scalar0 --glyphs sphere VisIVO Desktop VisIVOServer VisIVOWeb Interactive fast navigation NO Blocking function executes threads or separate processes VisI VO Library Grid and Databases: the BaSTI/FRANEC Example Key Concepts BaSTI (The Bag of Stellar Tracks and Isochrones) is a theoretical astrophysical catalogue that collects fundamental data sets involving stars formation and evolution. BaSTI is also a use-case for the Virtual Observatory, and a testbed for the definition of standards to access numerical simulations. To create and populate BaSTI, a large number of stellar evolutionary computations are necessary. Work already done to gridify FRANEC, the code used to feed BaSTI Creation of specific services in gLite Environment, used to submit both SMRs (Synthetic Model Runs) and FIRs (Full Isochrone Runs). - Desktop environment (to hide the complexity in configuring and submitting FRANEC tasks) - Computational Services (to manage the configuration and execution of FRANEC tasks) - Data Oriented Services (to manage simulated data) Services for gLite are designed and implemented in a modular way to make possible their reusability Activity currently in progress Integration in P-GRADE: A specialized dedicated portal has been built on top of P-GRADE. Modular Specialized Portal: components reused more times. - Added a page allowing users to insert values for all parameters of the job to run. - Automatic check of inserted data to verify their formal correctness. - New P-GRADE component (Java script) producing a P- GRADE compatible workflow starting from data provided by users. Improvements of FRANEC: the values of some fundamental parameters requested to run pipelines are automatically generated and fed to FRANEC through input files. Modification of some components of P-GRADE to automate the porting of jobs in Grid. The construction of appropriate P-GRADE compatible workflows in charge of the application. Workflows are submitted through the standard tools provided by P-GRADE. Next Steps Interfacing the BaSTI DB and the Grid and integrating BaSTI in P-GRADE. Repeated runs with fixed values for a set of K parameters and values ranging in predefined intervals for the remaining set of N-K parameters (e.g. run of X jobs with different values of the mass). Use of robot certificates to ease users in approaching and using the portal and the underlying Grid infrastructure. INAF is actively engaged in projects, both at national and international level, aimed at fostering the use of Grid technology for astronomers to carry out their everyday work, through the deployment of computing infrastructures and integration of domain-specific applications. At the European level, INAF currently plays a leading role in promoting a tight interoperability between the Grid and the International Astronomical Virtual Observatory. In this poster we present some important research projects extensively using the computational Grid resources. U. Becciani, A. Costa, P. Massimino, F. Vitello, C. Vuerli, S. Cassisi, P. Manzato, M. Molinaro, F. Pasian, A. Pietrinferni, M. Sponza, G. Taffoni - INAF – Istituto Nazionale di Astrofisica - ITALY

INAF - National Institute for Astrophysics

  • Upload
    vidal

  • View
    33

  • Download
    0

Embed Size (px)

DESCRIPTION

INAF - National Institute for Astrophysics. - PowerPoint PPT Presentation

Citation preview

Page 1: INAF - National Institute for Astrophysics

INAF - National Institute for Astrophysics

INAF - National Institute for Astrophysics

The National Institute for Astrophysics coordinates and participates in the Astronomy and Astrophysics (A&A) cluster in EGI, using extensive and large applications on the Grid. INAF is a partner of IGI (Italian Grid Initiative) and participates in its activities, with skilled personnel and with computational resources in several INAF local institutes.

VisIVO tools for Large Dataset Exploration on gLite GridThe Visualisation Interface to the Virtual Observatory (VisIVO) is a suite of software tools aimed at creating 3D customized views of many type of wide used data sets. Its peculiar characteristic is that there is no limit for what concerns the size of input tables containing data to be processed, thus they are able to support very large scale datasets (tens of Terabytes and more).

VisIVO is designed to create images and movies from user files. The first fundamental goal of porting VisIVO to the Grid is to have movies and images directly generated and stored on the Grid, using the internal arrays of a running code on worker nodes, and to reduce the overall time for movie production.A VisIVO library was designed from November to December 2010 and the development phase took place in the first four months of 2011. VisIVO library represent an important step in order to export the usage of the 3D visualization rendering and movie creation tools to many user communities. The user can set the environment variables to use the multithread or the multiprocess features. By default the multiprocess features are adopted. The user application may call several instances for VA_Importer, VA_Filter, VA_View and several threads or processes will be started simultaneously.

VisIVO may be compiled on a gLite system 3.1. In this case the GLITE option must be given when compiling the tool. We used lcg_util APIs: an high-level gLite data management APIs. They are able to provide a single interface to deal with Storage Elements and LFC. They actually provide they same functionality available throw the lcg_util command line tools (lcg-cr, lcg-cp, lcg-rep, lcg-del, etc). We used these functions to create a local copy of the table in the local WN.

User code linked with VisIVO Library

User code submitted to the Grid

User code Running on WnUser code produces images and

movies using VisIVO

Grid Catalogue

Monitor the data production during the run

--fformat votable /home/user/demo/vizier.xml.....--x x --y y --z z --color --colortable --colorscalar scalar0 --glyphs sphere

VisIVODesktop

VisIVOServer VisIVOWeb

Interactive fast navigation

NO Blocking function executes threads or separate processes

VisI VO Library

Grid and Databases: the BaSTI/FRANEC ExampleKey Concepts

BaSTI (The Bag of Stellar Tracks and Isochrones) is a theoretical astrophysical catalogue that collects fundamental data sets involving stars formation and evolution.BaSTI is also a use-case for the Virtual Observatory, and a testbed for the definition of standards to access numerical simulations. To create and populate BaSTI, a large number of stellar evolutionary computations are necessary.

Work already done to gridify FRANEC, the code used to feed BaSTI

Creation of specific services in gLite Environment, used to submit both SMRs (Synthetic Model Runs) and FIRs (Full Isochrone Runs).

- Desktop environment (to hide the complexity in configuring and submitting FRANEC tasks)- Computational Services (to manage the configuration and execution of FRANEC tasks)- Data Oriented Services (to manage simulated data)

Services for gLite are designed and implemented in a modular way to make possible their reusabilityActivity currently in progress

Integration in P-GRADE: A specialized dedicated portal has been built on top of P-GRADE.Modular Specialized Portal: components reused more times.

- Added a page allowing users to insert values for all parameters of the job to run.- Automatic check of inserted data to verify their formal correctness.- New P-GRADE component (Java script) producing a P-GRADE compatible workflow starting from data provided by users.

Improvements of FRANEC: the values of some fundamental parameters requested to run pipelines are automatically generated and fed to FRANEC through input files.Modification of some components of P-GRADE to automate the porting of jobs in Grid.The construction of appropriate P-GRADE compatible workflows in charge of the application.Workflows are submitted through the standard tools provided by P-GRADE.

Next Steps

Interfacing the BaSTI DB and the Grid and integrating BaSTI in P-GRADE.Repeated runs with fixed values for a set of K parameters and values ranging in predefined intervals for the remaining set of N-K parameters (e.g. run of X jobs with different values of the mass).Use of robot certificates to ease users in approaching and using the portal and the underlying Grid infrastructure.

INAF is actively engaged in projects, both at national and international level, aimed at fostering the use of Grid technology for astronomers to carry out their everyday work, through the deployment of computing infrastructures and integration of domain-specific applications. At the European level, INAF currently plays a leading role in promoting a tight interoperability between the Grid and the International Astronomical Virtual Observatory.

In this poster we present some important research projects extensively using the computational Grid resources.

U. B

eccia

ni, A

. Cos

ta, P

. Mas

simin

o, F

. Vite

llo, C

. Vue

rli, S

. Cas

sisi,

P. M

anza

to, M

. Mol

inar

o, F

. Pas

ian,

A. P

ietri

nfer

ni, M

. Spo

nza,

G. T

affo

ni

- IN

AF –

Istit

uto

Naz

iona

le d

i Ast

rofis

ica

- ITA

LY