Upload
spike
View
31
Download
3
Embed Size (px)
DESCRIPTION
Future GRID Strategy Beijing 7th of September 2004 (CNES, J.P.Antikidis). Reminders - GRID evolution and upcoming concepts Technological trends Report on WGISS action 17.4 The WGISS WAG experiments Further on activities. Exemple of distributed knowledge and process environment. - PowerPoint PPT Presentation
Citation preview
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Future GRID StrategyBeijing 7th of September 2004
(CNES, J.P.Antikidis)
- Reminders- GRID evolution and upcoming concepts-Technological trends-Report on WGISS action 17.4- The WGISS WAG experiments- Further on activities
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Exemple of distributed knowledge and process environment
Beijing 6th Sept 2004, CNES/J.P.Antikidis
IMAGINE….IMAGINE….
That you could plug your computer into the wall and have That you could plug your computer into the wall and have direct access to huge computing resources immediately, direct access to huge computing resources immediately,
just as you plug-in a lamp to get instant light.just as you plug-in a lamp to get instant light.This is the idea the Grid project is about to make into realityThis is the idea the Grid project is about to make into reality
Let’s quote Ian Foster and Carl Kesselman's book (The Grid): Let’s quote Ian Foster and Carl Kesselman's book (The Grid): ""A A computational Grid is a hardware and software computational Grid is a hardware and software infrastructure that provides dependable, consistent , pervasive infrastructure that provides dependable, consistent , pervasive and inexpensive access to high-end computational and inexpensive access to high-end computational capabilitiescapabilities".".
“The Power of Grids”
All data systems All data systems technologies are concernedtechnologies are concerned : :
* * NetworksNetworks
* * Operating systemsOperating systems
* * Large scale dataLarge scale data processing processing
* * Protected architecturesProtected architectures
* * Systems creation andSystems creation and programming programming
Beijing 6th Sept 2004, CNES/J.P.Antikidis
GRID
Classical GRID based
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Virtual machine defined by Middleware primitives
« Pseudo-machine »
GRID based activitiesGRID: Distributed machines organized by an Operating System GRID: Distributed machines organized by an Operating System
making individual hardware unvisiblemaking individual hardware unvisible
GRID is not an end by itself and coversGRID is not an end by itself and covers
#GRID techno developments#GRID techno developments#GRID based system concept#GRID based system concept
Present paradox with Middleware evolution: Present paradox with Middleware evolution: Make GRID invisible Make GRID invisible from users, so from users, so Who are the users (historical trend) ?:Who are the users (historical trend) ?:
# End users directly connected to GRIDs# End users directly connected to GRIDs
# System developers directly connected to GRID# System developers directly connected to GRID
# System developers willing to make use of GRID high level # System developers willing to make use of GRID high level services (Upperware High level O/S that may compile or emulate services (Upperware High level O/S that may compile or emulate needed systems)needed systems)
Strategic usersStrategic users thinking "thinking "differentlydifferently" because of the GRID advent" because of the GRID advent
Beijing 6th Sept 2004, CNES/J.P.Antikidis
The GRID Cook-book
Direct mapping Block module or API primitives
"High level O/S: Upperware
GRID evolution Creation of "Virtual machines" (Upperwares and HEAVEN)Creation of "Virtual machines" (Upperwares and HEAVEN)
Exchange power against flexibiliy (Upcoming upper-wares)Exchange power against flexibiliy (Upcoming upper-wares)
Create "Virtual machines at will"=> Virtual FactoriesCreate "Virtual machines at will"=> Virtual Factories
Several approaches:Several approaches: Data flow virtual descriptionsData flow virtual descriptions
"Virtual machine and processing Emulation" "Virtual machine and processing Emulation"
"Virtual layer of JAVA machines"Virtual layer of JAVA machines
Creation of "transparent" system connectivityCreation of "transparent" system connectivity
Why not leave Telecom and data transfers under O/S control ?Why not leave Telecom and data transfers under O/S control ?
"Distance is not anymore relevant=> Wide Aera Grids"Distance is not anymore relevant=> Wide Aera Grids
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Methodology trends
GRID+UpperWare (2002)Processing systems (1960)
SPICE simulators(1985)Electronic circuitry (1914)
INTERNET (1990)MINITEL Physical links(1970)
WINDOWS (1990)Formal programming (1948)
CATIA (1985) (virtual mockup)Blueprint plans (1850)
to « Virtual »From Real =>
"Hosting of Emulated Applications in a Virtual Environnement"
If it is possible to “instruct” a GRID to mimic (or If it is possible to “instruct” a GRID to mimic (or simulate) a computer system,simulate) a computer system,
Shouldn’t be possible to use this simulation as a Shouldn’t be possible to use this simulation as a replacement for the real system by directly running replacement for the real system by directly running the application software within the simulation itself?the application software within the simulation itself?
In short: If one can draw a computer system (that In short: If one can draw a computer system (that could be a large scale and widely distributed could be a large scale and widely distributed implementation) on a piece of paper , will it be implementation) on a piece of paper , will it be possible to create it with the help of a suitable high possible to create it with the help of a suitable high level operating system (called Upperware) level operating system (called Upperware)
In such a way it behave like a real system
If feasible , the consequences on our way to create operational and distributed collaborative system may change drastically and lead to a conceptual change comparable to Internet revolution ...
The HEAVEN paradigm
"The result produced by running an "emulator" is equivalent to the result produced by the real system."HEAVEN can "mimic" any required system
Instead of creating complex processing system, perhaps better describe them in a powerfull and fully flexible computing topology and make the resulting emulation running as a replacement for the real system.
The difficulties experienced by the KRELS are properly solved in a pure mathematic computer world !!!
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Virtual enabling facilitiesMost recent approach are connected to:-Compilers and mapping systems able to "tranlate" user requirement into GRIDbasic primitives (efficient but little flexibility and conventional way to treat the inormation
-Emulation of systems embedded into a general purpose supporting GRID creating " virtual machines"
For example the Emulated type approach allows to inject conventional SW and play with system without changing any users'habits (in therory no-one can make the difference in between a Virtual based applicaion and a conventional implementation, but the development cost and effort is several order of magnitude lower
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Computers (1950)+ network (1960)=> distributed processing + Packet transmission(1980) + UNIX/WINDOWS O/S(1980)+Mosaic high
level interface (1990) =Internet revolution
Computers(1950)+networks+middleware (1990)= Grid (1995+Upperware (2002)= Virtual defintion and running of processing systems (2005 ….)=
=> Full virtual data management factories (evolution or revolution ?)Advanced Upperwares like HEAVEN is to GRID the equivalent of an advanced O/S like Windows vis à vis a PC-BIOS (inside the machine everything is 8086 protocols but nobody cares anymore).To day using GRID on a direct fashion (trough middleware primitives) is equivalent to code an applicaion in machine language.
From stone-age to modern time
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Technology trends and goals
Machines running and software implementaion as general purpose services fully shared on an invisible fashion among all operators
Information, processing, organization of system fully virtual (you just pay for GRID use and access to data sets)
Full analogy with Telecomm:You pay for time use of a system, not anymore for individual components uses.
Data use will not anymore associated to an image or a data set, it could be associated to its use
Applications systems created by the users themselves
Beijing 6th Sept 2004, CNES/J.P.Antikidis
"The WAG Experiments"
Response to action 17.25
-"J.P.Antikidis with input from J.Hielkema and Yonsock "J.P.Antikidis with input from J.Hielkema and Yonsock to prepare a paper on the system process for Wide Aera to prepare a paper on the system process for Wide Aera Grid (WAG)"Grid (WAG)"- Meeting with FAO (end of August)Meeting with FAO (end of August)- Coordination with ESA/ESRIN-> MMECoordination with ESA/ESRIN-> MME
(Meris-Modis Experiment)(Meris-Modis Experiment)- CNES organized effort on GRID experimentCNES organized effort on GRID experiment
=>> Proposal to hold demonstrative test based on "real =>> Proposal to hold demonstrative test based on "real size case" illustrating the capability of WAGsize case" illustrating the capability of WAG
Virtual handling of information
GRIDs main characteristics:GRIDs main characteristics:-Flexible topologyFlexible topology-Data flow handlingData flow handling-Computing PowerComputing Power-Distributed processingDistributed processing
=>>Let's add an upper layer "upperware" able to =>>Let's add an upper layer "upperware" able to submit user's jobs to the system= Virtual submit user's jobs to the system= Virtual handling of informationhandling of information
An illustration of Virtual storage handling
ConventionalConventional::
Locate the information (catalogues etc..)Locate the information (catalogues etc..)
Get hold of the data (media, access costs, transmission & delays)Get hold of the data (media, access costs, transmission & delays)
Process the informationProcess the information
Transfert the result to suitable placeTransfert the result to suitable place
WAG based conceptWAG based concept::
Create a "Light" Grid connecting a few centersCreate a "Light" Grid connecting a few centers
Make their archive accessible from this GRIDMake their archive accessible from this GRID
Enable the creation of jobs submitted to this GRIDEnable the creation of jobs submitted to this GRID
=>>Then could be demontrated the feasibility of a virtual factory =>>Then could be demontrated the feasibility of a virtual factory able to dematerialize the handling of informationable to dematerialize the handling of information
Beijing 6th Sept 2004, CNES/J.P.Antikidis
GRIDS speciesALL GRID ARE NO EQUIVALENT FROM THE ALL GRID ARE NO EQUIVALENT FROM THE
EO-SCIENCE PERSPECTIVEEO-SCIENCE PERSPECTIVE
- Macro-Grids (ganging of processors)(dynamic hanling of HW resources, eg multi-CPU machines) ~ meters (dynamic hanling of HW resources, eg multi-CPU machines) ~ meters (efficiency)(efficiency)
- Local Grids (Networked computers)with Grapes of computers (local GRIDS, eg:CERN machines) ~decameters with Grapes of computers (local GRIDS, eg:CERN machines) ~decameters (power)(power)
Wide Aera Grids(Widely distributed systems) worldwide basis (integration of services)worldwide basis (integration of services)
"WAG" concept
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Virtual Systems internal mechanism
Physical layout(general purpose)
Middleware handling(Basic O/S)
Upperware(high level O/S)
CAD “like” design(Virtual factory)
“Virtual layout”
“Real interface”
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Induced conceptual changes
No more knowledge of information or/and data-bank physical location (an archive is just known to be accessible trough its naming)No more need to describe the physical connections in between different locations (since fully supported by the underlying GRID) making in theory any world-wide distributed systems easy to implement.No more physical handling of information sets. This could trigger completly new merchandising concepts (for ex. come close to Telecomm recharge)No need to know the mapping of user ’software enabling virtual service to be easealy created and offered to any-one on an autonomous fashion (virtual services)
Beijing 6th Sept 2004, CNES/J.P.Antikidis
W.A.GRID based
Data providers
System designer
User agent
WAG
User Descr.
Data
User data
Real flow
"Virtual elements
GRID
A start : The “Meta Computing”
Process Process localisation is not localisation is not anymore relevantanymore relevant
Job control is not Job control is not fully machine fully machine independant and independant and special care is special care is still needed wih still needed wih respect to respect to Application Application mappingmappingll
A “Proto”-WAG experiment:GRID5000
Grid'5000 is a research effort Grid'5000 is a research effort
developping a large scale nation developping a large scale nation wide infrastructure for Grid wide infrastructure for Grid research.research.
The current plans are to assemble a The current plans are to assemble a physical platform featuring 8 physical platform featuring 8 clusters, each with 100 to a clusters, each with 100 to a thousand PCs, connected by the thousand PCs, connected by the Renater Education and Research Renater Education and Research Network. 7 sites are already Network. 7 sites are already selected. A 8th one sould be selected selected. A 8th one sould be selected soon. All clusters will be connected soon. All clusters will be connected to Renater at 1 Gb/s (10 Gb/s is to Renater at 1 Gb/s (10 Gb/s is expected in near future).expected in near future).
This high collaborative research effort is funded by the French ministry of Education and Research, INRIA, CNRS, the Universities of all sites and some regional councils. 10 laboratories are involved, nation wide, in the objective of providing the community of Grid researchers a test-bed allowing experiments in all the software layers between the network protocols up to the applications:
Beijing 6th Sept 2004, CNES/J.P.Antikidis
fMODISMERIS
Experiment(MME)
VirtualFactory
Experiment(VFE)
……
WGISS WAG Experiments
-Archives are not "physically" but logically" Archives are not "physically" but logically" describeddescribed-Processes are described as "virtual actions"Processes are described as "virtual actions"- The underlying HW layout is managed by an The underlying HW layout is managed by an underlying Middleware/Upperwareunderlying Middleware/Upperware
Beijing 6th Sept 2004, CNES/J.P.Antikidis
Near future plans: “Earth Science Grid on Demand” Within CEOSWithin CEOS: :
Share data and resources from the different agencies Share data and resources from the different agencies Provide secure and controlled access to the different Provide secure and controlled access to the different
data storage elements data storage elements Share computer resources specific modules Share computer resources specific modules Test and evaluate new processing data fusion Test and evaluate new processing data fusion
algorithmsalgorithms
Specific case: MERIS and MODIS vegetation Specific case: MERIS and MODIS vegetation productsproducts
WAG Experiment
The Virtual Factory ExperimentThe Virtual Factory Experiment Connect several resource center in a WAGConnect several resource center in a WAGMake Process server availableMake Process server availableCreate a virtual distribution schemeCreate a virtual distribution schemeAllow the projection by users of needed processAllow the projection by users of needed processPotential partners: Data and GRID providers Potential partners: Data and GRID providers (Europe+others), Remote job submitter and users (DC (Europe+others), Remote job submitter and users (DC representatives like FAO and UNEP )representatives like FAO and UNEP )
WAG Experiments & WGISS layout -Identification of a potential "task" called WAG Identification of a potential "task" called WAG experiment covering several demonstratives "test experiment covering several demonstratives "test case"case"-Task group composed of test case participants:Task group composed of test case participants:
ESA,CNES,NASA,FAO,UNEP etc…ESA,CNES,NASA,FAO,UNEP etc…
- Activity being blessed by WGISS18 ?- Activity being blessed by WGISS18 ?
- Consolidated proposals with respect to the two - Consolidated proposals with respect to the two first experiments headed by ESA (MME) and first experiments headed by ESA (MME) and CNES (VFE) to be established during CNES (VFE) to be established during WGISS19/20 time frameWGISS19/20 time frame
Beijing 6th Sept 2004, CNES/J.P.Antikidis