View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Workshop on Grid Utilization - IAS – 20/10/2005 - 2
• Long-term partner for our customers in design, integration and operations of critical applications, systems and infrastructures
• Focus on applications at the heart of our customers' business : expertise, commitment, innovation, sense of service and ability to manage large and complex projects
+3,300 employees (70% engineers)
Headquarters and Regional offices in France6 subsidiaries in EuropeSubsidiaries in Canada, USA, Brazil and Chile
2004 Consolidated Sales : € 348 Millions
Workshop on Grid Utilization - IAS – 20/10/2005 - 3
Intelligent Transport Systems
Cross-Disciplinary Offers
Banking and Finance
Energy, Utilities and Telecom
Public Administrations
Defence and Military Aviation
Aeronautics and Automotive
Space and Civil Aviation
Critical Systems Integration
Outsourcing &Transitioning IS
NetworkServices
Industrial & CriticalApplications
Critical Real Time software
Mission and Control Centres
Simulation and Scientific Computing
Technical Information Systems
(PLM, EDMS, EAI, Web-services, Virtual
Reality)
Security & Business Continuity
Subsidiaries•SCOT
•Germany•Canada•Romania
Air Traffic Control Centre
Simulator
CommunicationSystems
IntelligenceC4I
Logistic InformationSystems
Toll Systems &Contact less toll
collection
Traffic Management
Subsidiaries•United States
• Chile• Croatia
DistributedSystems
Networks
End-to-End Production
DataMigration
Systems Conversion
Application Maintenance
Architecture and Design
Integration and Deployment
Maintenance and Supervision
Security
Digital Video Surveillance
Application Management / Security / Linux & Open-source Software / Grid Computing
Infrastructure ServicesSystems and Applications
Business skills adapted to our clients needs
Workshop on Grid Utilization - IAS – 20/10/2005 - 5
Computing and storage facilities
Security and confidentiality
Capacity to use a set of dedicated applications closed to their industrial needs
Costs reduction by mutualisation and computing on demand
Externalization of IT tasks
‘User friendly’ interfaces to HPC infrastructures
Business model matching to their financial capacities
Industrial Applications driven Grids
RUGBI : To SMEs of Biotechnology Industry
Openplast : To SMEs of Polymer Processing Industry
SMEs Needs and Challenges
Workshop on Grid Utilization - IAS – 20/10/2005 - 6
Application driven Grid Challenges• Needs
• Non computing specialist users.
• Light Client with grid frontier (portal).
• Information System for user request, management and exploitation.
• Technologies Web compliant.
• Quality Of Services (QoS)• Transparency.
• Security / Confidentiality.
• Performance / Permanence.
• Easy business oriented increase and scalability.
• Interoperability, Evolution compliant, Portability.
• Administration / Exploitation / Maintenance.
• Distributed Architecture.
• Open source and Open file format solutions.
• Initial state of the art• DataGrid, E-Toile, GRASP, …
Workshop on Grid Utilization - IAS – 20/10/2005 - 7
Two Grid projects gathering:
OpenplastRUGBI
Funded by the French Ministry of Research
IT Services company
Software Editor
Business oriented interface
Academic institution
Infrastructure provider
Workshop on Grid Utilization - IAS – 20/10/2005 - 8
Academics & IT services company :
Computing cluster MPI
Academics & IT services company : Storage bay
IT services company:
Controllerserver
IT services company: Exploitation
server
Actors driven Architecture
Client
SMEs :End - User
Technological park & Editors & Academics :
Web PortalBusiness oriented
Workshop on Grid Utilization - IAS – 20/10/2005 - 9
High speednetwork
ECP [Paris]PEP [Oyonnax]
CS SI [Grenoble]
CC-IN2P3 [Lyon]
Client
INSA [Oyonnax]LPC
[Clermont Ferrand]
Grids Sites Infrastructures
Openplast RUGBI
Controller server
Web Portal serverComputing Cluster
Storage BayExploitation Server
CC-IN2P3 [Lyon]
CS SI [Grenoble]
High speednetworks
ECP [Paris]
Client
IBCP [Lyon]
Client
Academicinstitutions
BioPôle [Clermont limagne]
Workshop on Grid Utilization - IAS – 20/10/2005 - 10
The grid portal must provide a user friendly interface for grid management, exploitation and easy use.
User management
Resources management
Sites management
Workflow managementFiles management
Exploitation services
Grid management
Jobs management
Grid
Portal
Workshop on Grid Utilization - IAS – 20/10/2005 - 11
Resources management services :
• Applications (commercial or not): • Creation.
• Deployment.
• Configuration.
• Update.
• Databases (commercial or not): • Creation.
• Deployment.
• Configuration.
• Update.
• Storage resources with specific roles:• Transit (all uploaded data) on portal.
• Repository (all permanent data) on storage element.
• Working (for job submission) on computing element.
Workshop on Grid Utilization - IAS – 20/10/2005 - 12
CS GRID Controller : multi-layer architecture
Operating System
Batch Manager
Globus
Grid Services
ControllerInformation System Data
DBMSXML-DB XML Files
Soft & DBdeployment
User Resources Rights Execution/Workflow
Location/Allocation
Filesmanagement
Exploitation
Web Portal
Bu
sin
ess o
rien
ted
ap
plicati
on
s
Bu
sin
ess o
rien
ted
ap
plicati
on
s
Generic layers Specific layers
Controllers provide services to manage users, resources and users rights on resources.
Controllers provide allocation services to allocate jobs execution.
Controllers provide exploitation services : accounting, monitoring, …
Workshop on Grid Utilization - IAS – 20/10/2005 - 13
• Globus 2.4, 3.2, and now 4.0
• Java (servlet, CoG)
• Web Services SOAP
• OpenSource schedulers (PBS, Condor, …)
• Exchange format : XML
• Information System hosted on DBMS Oracle / Mysql / Xindice wrappers
Technological choices
Workshop on Grid Utilization - IAS – 20/10/2005 - 14
Computing
Storage
Controller
Exploitation
Portal
Client
• Client Web• Client SOAP• Client CoG Globus GridFtp, MyProxy• Client Mail
•Web Server• Client WS-SOAP•GridFtp Server
• WS-SOAP Server• CoG Globus Gram, GridFtp, MyProxy Clients • SGDB-XML Server
• WS-SOAP Server • MyProxy Server • NTP Server • CA Server • Mail Client
• GridFtp Server • GRAM Server • SOAP Server
• GridFtp Server
FIREWALLSOAP [8080]MyProxy [7521]
FIREWALLGRAM-SOAP [8080]GridFtp [2811]
FIREWALLGRAM-SOAP [8080]GridFtp [2811]
DMZ
FIREWALLSOAP [8080]
FIREWALLSOAP [8080]
FIREWALLGridFtp [2811]HTTPS [8443]
Grid internal Communication
Client external Communication
Portal-Controller Communication
Grid Network Architecture
Workshop on Grid Utilization - IAS – 20/10/2005 - 15 Internet provider Storage and Computing Network
Intranet CS Network
DMZ
CS Node Network Architecture compliant with internal and external grids
Workshop on Grid Utilization - IAS – 20/10/2005 - 16
Computing cluster MPI
Storagebay
Computing cluster MPI
StoragebayController
server
Exploitationserver
Site A
Site B
User portal
Connection
Authorization
Authentication
proxy creation
certificate creation
Grid basic job submission principles
Web Portalserver
Workshop on Grid Utilization - IAS – 20/10/2005 - 17
Computing cluster MPI
Storagebay
Site B
User portal
File
Computing cluster MPI
StoragebayController
server
Exploitationserver
Site A
File
Job submission
Workflowsettings
Fileupload
Web Portalserver
Workflowgeneration
Workflowexecution
File transfer
File
Jobexecution
Grid basic job submission principles
Workshop on Grid Utilization - IAS – 20/10/2005 - 18
Computing cluster MPI
Storagebay
Site B
User portal
Computing cluster MPI
StoragebayController
server
Exploitationserver
Site A
Web Portalserver
Workflowexecution
File
Jobexecution
result
Files transfer to user spaceWorflow engine informs user
Updates portal/
send mail
Grid basic job submission principles
Workshop on Grid Utilization - IAS – 20/10/2005 - 20
Examples : Installation packages Controller configuration
Intuitive Grid Portal/Controller installation & configuration
Workshop on Grid Utilization - IAS – 20/10/2005 - 21
Examples : Grid-client Files Management User management
User friendly interfaces
Workshop on Grid Utilization - IAS – 20/10/2005 - 22
Exploiting and Monitoring• Subscription forms
• Statistical use diagrams
Workshop on Grid Utilization - IAS – 20/10/2005 - 24
RUGBI Tools and databases
Automatically Updated Databases
Protein knowledgebaseSwissprotTrembl
3D protein structure dataPDB
Encyclopedia of genes and genomesKEGG
nucleotide sequence database EMBL
compounds diversity set NCI
Tools
Sequence alignmentBlastFasta
Multiple sequence alignmentClustalw
Protein 2D structure prediction PredatorSympa96GorIV…
Docking of compounds to proteinsAutodock
Metabolic Pathway Analysis
Workshop on Grid Utilization - IAS – 20/10/2005 - 26
Platform for polymer injection molding simulation• Project management : case-studies oriented
• Allows to specify all the stages of polymers injection modeling
• Qualifying and Validating each stage of the process
CADCAD
ProjectProject
MeshMesh
SubprojectSubproject
Datasettings
Datasettings
StudyStudy
MoldingMolding
CaseCase
AnalysisAnalysis DefectsDefectsStudy view
Project view
• Data enrichment
Mesh Soft.
Mesh Soft.
SolverSolver Post-treatment
Post-treatment
CAD +Mesh
Plast3DPlast2DPlastTherm3DPlastPart2D
PamViewPamRTM
Geomesh
CAD +Mesh +DS [LC, IC, FC]
PlastDefects
XML
Data settings
Data settings AnalysisAnalysis
PlastPre
CAD CAD +Mesh +DS +Raw results
CAD +Mesh +MED +Raw res. +Analysed res.
CAD +Mesh +DS + Raw res. +Analysed res. +Post-treated res.
CATIA
DSDS : : Data Data settingssettings
XMLXML XMLXMLXML XML XML
Workshop on Grid Utilization - IAS – 20/10/2005 - 27
Study Framework
Prototype Openplast client modules
Post-treatment framework
Workshop on Grid Utilization - IAS – 20/10/2005 - 28
• Applied grids for SMEs :• Security based on Globus authentication by proxy and certificate.
• Confidentiality managed by a strict use of user directories (temporary, working, storage,…) based on user rights.
• Exploitation and Accounting facilities for different user type.
• Easy of use for non grid specialists.
• Basic set of applications (client and grid side).
• Expected Grid End-Users• RUGBI
• IBCP portal gets daily 2000 connections that will be transferred on RUGBI nodes.• Different SMEs from Clermont Limagne BioPole and others.
• Openplast• Polymer processing industry from which 70% are SMEs.• European Polymer Institute.
Conclusion
Workshop on Grid Utilization - IAS – 20/10/2005 - 29
• Starting point: the SMEs real needs
• Business Interface partners
• Openplast : European Polymer Institute
• RUGBI : BioPôle Clermont Limagne.
• In progress: • RUGBI: study market with Lyon Biopôle, Arteb, Théogone …• Openplast: partnership with PEP.
• IT service company (IN2P3, CS).
• Academic institutions for their up-to-date research knowledge.
• Software editors: Openplast: ESI, Fluent, … & RUGBI: Accelrys, Tripos, …
• Infrastructure: extend from a regional level to national
• RUGBI: Theogone and different biopoles and interoperability with future national french bioinformatics grid.
• Openplast: HPC pole - Iles de France (IOLS).
• European: EGEE2 (functional test OK with GT2.4, and interoperability with Glite in progress).
For a good industrial accomplishment
Workshop on Grid Utilization - IAS – 20/10/2005 - 33
USER_IDUSER_TYPE
USER_IDLOGINPWD
CERTIFICATION_SUBJECTCERTIFICATION_CREATION_DATE
CERTIFICATION_END_DATECERTIFICATION_RENEW_DATE
CERTIFICATION_AUTHORITYCERTIFICATION_TYPE
STREETCITY
ZIPCODELAST_NAMEFIRST_NAME
CREATION_DATEEND_DATE
STATE
USER_IDDESCRIPTION
NAMECREATION_DATE
END_DATESTATE
1..*
0..*
RIGHTS
USER_IDRESOURCE_ID
RWXSTATE
PERIODPRIORITY
QUOTA
RESOURCES
RESOURCE_IDSITE_ID
RESOURCE_TYPE
0..*
0..*
RESOURCE_IDSIZES
DESCRIPTIONFORMAT
DATA_TYPE
RESOURCE_IDNAME
DESCRIPTIONVERSIONEDITOR
ADMINISTRATOR_NAMEXML_FILE_NAME
RESOURCE_IDNAMESIZESURL
DESCRIPTIONVERSIONFORMAT
ADMINISTRATOR_NAMEXML_FILE_NAME
DATAS SOFTWAREDB
SITE
NAMECITY
SITE_IDCOMPANY
DESCRIPTIONXML_FILE_NAME
ADMINISTRATOR_ID
SITE
NAMECITY
SITE_IDCOMPANY
DESCRIPTIONXML_FILE_NAME
ADMINISTRATOR_ID
0..*
10..*
USERS
1..*
1
0..*
GROUPS CERTIFIED
Manage users, groups and rights on resources.
Manage logical resources : softwares and databases.
Manage physical resources : Computing / Storage elements, Portals …
Information System 1/3
Workshop on Grid Utilization - IAS – 20/10/2005 - 34
Database updating RUGBI service allows :• Releases updates
• Versions updates
• Versions management
• Versions deployment
FTPserver
Available files list
XML FILE
Comparison
Metadata(names, date)
Available files list
XML FILE
Missing files
Download
- Waiting end of jobs on old database- Deletion of the old database
- Replication- Registration
Update
Workshop on Grid Utilization - IAS – 20/10/2005 - 35
Computing cluster MPI
StoragebayMaster
Controllerserver
Exploitationserver
Site B
Web Portalserver
Computing cluster MPI
Storagebay
SlaveController
server
Site A
Web Portalserver
datamodification
Computing cluster MPI
Storagebay
SlaveController
server
Site C
Web Portalserver
User portal
dataModificationnotification
dataModificationnotification
dataupdates
Grid information system update mechanisms
Workshop on Grid Utilization - IAS – 20/10/2005 - 36
Computing cluster MPI
StoragebayMaster
Controllerserver
Exploitationserver
Site B
Web Portalserver
Computing cluster MPI
Storagebay
SlaveController
server
Site A
Web Portalserver
Computing cluster MPI
Storagebay
SlaveController
server
Site C
Web Portalserver
User portal
dataupdate
notification
dataupdates
dataupdates
Grid information system update mechanisms
Workshop on Grid Utilization - IAS – 20/10/2005 - 37
Controller
Interface<<DataRequestor >>
MySQL
XIndice
Wrapper
MySQLRequestor
Wrapper
XIndiceRequestor
DataRequestorFactory
Dump.xml
Requests Backup
Restore
Wrappers :
MySQL, Oracle, PostGres, XIndice compliant.
Backup / Restore throw XML files driven by the master controller.
Information System 3/3
Workshop on Grid Utilization - IAS – 20/10/2005 - 38
GRID
Workflow
EngineApplication developer
Submit jobs, transferfiles, invoke grid services, receive notification events, etc.
While:- Monitoring execution- Handling errors- Recovering workflow failures- Freeing allocated resources- etc.
End-user
Chooseapplications, input files, arguments,etc.
WebPortal
s+
Controller
job1
job2
jobs
executionenvironment
GRID : End-user interface / Application developer interface
Workshop on Grid Utilization - IAS – 20/10/2005 - 39
<graph wf:NAME="graph-8433" xmlns:wf="http://www.in2p3.fr/workflow/language" xmlns:x0="http://www.w3.org/2001/XMLSchema"> <wf:job wf:NAME="job-8433" wf:contact="sirius.si.c-s.fr:2119/jobmanager-fork" wf:isBatchJob="false" wf:provider="GT2">
<wf:executable wf:shell="/bin/sh" wf:stageIn="true" wf:tmp="/grid/transit/user/1/projects/8433/"> <![CDATA[/grid/software/BLAST/2.4/bin/blastall
-p blastn -i $INPUT -d /grid/db/ecoli/ecoli.aa -o $OUTPUT -v 500 > $OUTPUT]]></wf:executable><wf:sandbox wf:intermediary="gsiftp://sirius.si.c-s.fr:2811/grid/vo/CS-VO/cs-user1/projects"> <wf:input wf:VAR="INPUT" wf:source="file:///grid/transit/user/1/projects/8433/input.in"
wf:destinationdir="$HOME/tmp/8433" > <wf:output wf:VAR="OUTPUT" wf:destination="file:///grid/transit/user/1/projects/8433/output.out"
wf:sourcedir="$HOME/tmp/8433"/></wf:sandbox>
</wf:job> <wf:mail wf:NAME="mail-8433" wf:contact="mail.grid.fr" wf:depends="job-8433"
wf:from="[email protected]" wf:subject="your job 8433 is finished." wf:to="[email protected]"> <wf:message> <![CDATA[ Your results are on your transit space on CS grid portal CS ... ]]></wf:message> </wf:mail></graph>
Workflow used spacesPortal transient space (input and output) /grid/transit/user/1/projects/8433Temporary CE space gsiftp://sirius.si.c-s.fr:2811/grid/vo/CS-VO/cs-user1/projects/Temporary Working node space $HOME/tmp/8433CE software location /grid/software/BLAST/2.4/bin/blastall :CE database location /grid/db/ecoli/ecoli.aa
Execution sequence executed by the workflow engine
Execution sequence example