Upload
archibald-matthews
View
226
Download
1
Embed Size (px)
Citation preview
OGCEOGCEConsortium
Outline
• The Portal Challenge
• The Solution: – The Open Grid Computing Environments
Consortium (OGCE)
• The OGCE Portal Architecture
• Application Portal Examples
• NMI OGCE Plans
OGCEOGCEConsortium
Por
tal U
ser
Inte
rfac
e Grid ResourceBroker Service
Grid and Web
Protocols
Informationand
Data Services
DatabaseService Database
HPCor
Compute Cluster
Grid InformationServices, SRB
PortalClientStub
PortalClientStub
PortalClientStub
JDBC,Local, orRemote
Connection
Three-Tiered Architecture
Three-tiered architecture is accepted standard for accessing Grid and other services
OGCEOGCEConsortium
Portal are Gateways to Grid
• Early Grid Portals were great success:– Applications oriented,– Provided scientists with some useful tools– Encapsulated some complexities.
• But they were:– All “stovepipe” solutions.
• Very little reusable code and infrastructure
• Nothing could be shared between portals
– Many ignored emerging architecture of the Grid• A framework based on distributed ubiquitous services
OGCEOGCEConsortium
Impact of Distributed Grid and Web Services
Resource layer1000s of PCs ->massive supercomputers
Grid Portals
Launch, configureAnd control Application Interface
Workflow service
App InstanceApp InstanceApp Instance
SecuritySecurity
Data ManagementService
Data ManagementService
AccountingService
AccountingService
LoggingLogging
Event/MesgService
Event/MesgService
PolicyPolicy
Administration& Monitoring
Administration& Monitoring
Grid OrchestrationGrid Orchestration
Registries andName binding
Registries andName binding
Reservations And SchedulingReservations
And Scheduling
Open Grid Service Architecture Layer
Open Grid Service Infrastructure (web service component model)
Online instruments
(slide by Dennis Gannon)
OGCEOGCEConsortium
Advantage of Portlet Architecture
• Portlet/component approach very compatible with Web/Grid services model
• Each Grid service can be associated with a unique portlet– Very easy to add new services– Many different Groups can contribute portlets which can be
plugged into a portal.• Currently Indiana, Argonne, Michigan, NCSA,Texas + GridSphere
portal group following this model. Most likely others as well.– Jetspeed is Apache standard. Basis for IBM WebSphere portal.
Also Oracle, Sun and others. • Now a Java Standard (waiting for JSR 168)
• User can select and configure portlets he/she wishes to use– Selection becomes part of persistent context
OGCEOGCEConsortium
The Next Generation Grid Portal
• Building on Standard Technologies– Portlet Design (JSR-168) IBM, Oracle, Sun, BEA, Apache– Grid standards: Java CoG, Web/Grid Services– Web server: JetSpeed (open source)
• User configurable, Service Oriented– Philosophy: The Portal is a gateway to distributed Grid and Web
Services• With common API, portlets can be exchanged, interoperate
Portal Server
MyProxyServer
MetadataDirectoryService(s)
Directory& indexServices
ApplicationFactoryServices
Messagingand group
collaboration
Event andlogging
Services
OGCEOGCEConsortium
User Requirements Have Not Changed
• Users & Clients are:– Portal clients/users– Portal developers
• Security– Single sign-on, authentication to all grid resources
• Certificate based / uses MyProxy– VO management
• File Management– Grid FTP– File staging– Metadata management
• Remote Job Management– Job submission, scheduling, and tracking. Log and output mgmt.– Workflow management tools
OGCEOGCEConsortium
User Requirements
• Information Services– Grid resource and status information directories– Data and metadata catalogs – Personal and VO metadata and service index/directories– Database Interfaces
• Application Interfaces– Configure and launch workflow from domain specific interface
• Access to Collaboration Tools– VO collaboration. Synchronous and asynchronous– Access Grid venues
• Developers need common API
OGCEOGCEConsortium
Open Grid Computing Environments (OGCE) Consortium
• Motivation:– Emerging technologies impacting portal developers:
• Grid/Web services; ubiquitous database use, XMl
– Lessons learned from GCE-RG and other projects (GridPort/HotPage, Alliance Portal, DOE Components, NEESGrid, Gateway, GRaDS, others)
– Multiple projects repeating similar tasks, work, development;
• OGCE established in Fall 2003 to foster collaborations and sharable components with portal developers worldwide.
OGCEOGCEConsortium
OGCE
• Funded by:– NSF NMI: software releases will be part of NMI
and synchronized with GRIDS releases– Also: NCSA Alliance, NPACI, DOE, NASA,
others
• Release Schedule:– First Release: 11-15-03– Next Release: April, 2004
• Website: http://www.ogce.org
OGCEOGCEConsortium
OGCE Tasks/Goals
• Development of interoperable Grid portal architecture – Define standards for common API’s between portlet
service and remote services• Establishment of a Grid Portal Collaboratory
– Repository of portlet and portal service components• An online forum for developers of Grid Portals• The building of reusable portal components:
– Can be integrated in a common portal container system• Contribute to standards organizations:
– NSF NMI program software releases– GGF if appropriate or applicable
OGCEOGCEConsortium
OGCE Initial Architecture
Po
rta
l Local
Portlets
Teamlets
Proxy Portlets
JetspeedInternalServices
JavaCOGAPI
Java CoGKit
Grid Services
GridProtocols
GRAM,MDS-LDADMyProxy
ServiceAPI
CHEFServices
Rem
ote
Inte
rfac
es CoG
StubsHTTP
Grid Services
Other ServicesSOAP
Initial architecture aggregates multiple services into a single portal using portlet containers
OGCEOGCEConsortium
OGCE Initial Arch: Features & Capabilities
• OGCE approach encompasses multiple portal development systems:– Preserving development investments
• Central portal system (Jetspeed) acts as a central portal container:– Portal responsible for managing the lifecycle of pluggable portal
components, or portlets.
• Grid/computational portlets must provide services on the behalf of the user:– Not just a collection of links/static pages
• User view (from the left) is of a single portal composed of portlets.
OGCEOGCEConsortium
Supports Aggregation of Diverse Tasks
• Portlets provide a uniform management interface to the portal (on the left) connect to heterogeneous set of APIs on the right. – First horizontal thread
• Portlets interact with web interfaces written in other languages such as Perl, Python, PHP, ColdFusion, etc. – Architecture for bridged support is depicted in the second
horizontal thread • Examples:
– Locally running Myproxy Grid portlet uses Java CoG kit to interact with remote Grid services
– Job Scheduler portlet uses GridPort Toolkit to submit job via Community Scheduling Frameworks (CSF) service
– Teamlets (sharable portlets) share collaboration and Grid services, such as calendars, message boards, file services.
OGCEOGCEConsortium
OGCE - Revised Architecture
• Revised portal architecture provides a unified interface for portal services.• Supports aggregation of services in backend
Po
rtal
Portletsand
Teamlets
JetspeedInternal Services
ServiceAPI
GridProtocolsGrid
ServiceStubs
RemoteContentServices
Remote Content ServersHTTP
GridServices
Java
Co
G K
it
LocalPortal
Services
OGCEOGCEConsortium
Features
• All services will employ a single (generic) service API that allows them to be deployed and accessed in a uniform fashion.
• Developer then chooses (during portal configuration) the implementations of particular services.
• Services may be local, may access remote sites via Grid protocols, or may access remote content, such as web interfaces written in Perl.
• OGCE Twist:– Entire portal will auto-install and include all required
software components
OGCEOGCEConsortium
Key Technologies????
• Core Portal Framework: – CHEF/JetSpeed (U. Mich)
• Core Grid Technologies:– JavaCoG (provides Globus interface)– GridPort Toolkit (aggregates JavaCoG, and
other grid services)
OGCEOGCEConsortium
TeraGrid Portal • Access Point for TeraGrid
Services• Version 0: Collecting Initial
Services– Public Information about
Resources– Private Information for the
developers.
• Version 1: User centered portal– HotPage/GridPort style access to
user accounts, credentials, job submission & management.
• Version 2: Portals for Science Collaborations (Q3 2004)– Shared spaces, whiteboards, AG
access, group authorization, shared application services
OGCEOGCEConsortium
TeraGrid Integrated Architecture
Diagram demonstrates how existing software projects (such as GridPort) can be adapted to support NMI Portals software system
Po
rtal
Portletsand
Teamlets
JetspeedInternal Services
GridServiceStubs
RemoteContentServices
Remote Content ServersHTTP
GridServices
JavaCoG Kit
LocalPortal
Services
ServiceAPI
Gri
dP
ort
To
olk
it
WebServices
OGCEOGCEConsortium
Create job
description
Create job
description
Store andrecordresults
Store andrecordresults
Generateand storederived
products
Generateand storederived
products
monitoringmonitoring
myMEAD personal
informationspace
myMEAD personal
informationspace
Broker ServiceBroker Service
Teragriddata service
Teragriddata service
TeragridTeragridGlobus Toolkit 3.0Globus Toolkit 3.0
GridFTPGridFTP MetaCatalogService(MCS)MetaCatalogService(MCS)
NEESgrid Meta-Catalog service
NEESgrid Meta-Catalog service
Storage Resource Broker (SRB)
Storage Resource Broker (SRB)
Gather data files, resources
Gather data files, resources
Execute applicationExecute
application
Need ‘x’put to ‘l’by time y:00
Need tostore ‘y’ and ‘z’
Provides contextto resolve ‘x’, ‘y’, ’z’
Retrieve customized run description templatefrom data managementservices Monitoring results captured in event
services persistent channel. myMEADrecords address of service and channel
Delayed invocationof GridFTP
Teragrid Data Service-single point of contact to server-side data servers
Alliance scientific portal
OGCEOGCEConsortium
DOE Fusion Grid Portal
• Portal Development on the DOE SciDAC Grid– Initially, Fusion Grid
• Multi-site collaborative project: – TACC, Indiana, SDSC,
General Atomics
• Developing interoperable portal and Web services
• Portal architecture based on NMI Portal software
OGCEOGCEConsortium
CHEF - NEESGrid
• NEES Tools– Data Viewers (Live/Stored)– Telepresense– Video as Data– Electronic Notebook– Data/Metadata Repository
• Grid Technologies– Grid sign on using myproxy– Grid computational portal– GridFTP– ..Many more
OGCEOGCEConsortium
OGCE Portal System
System Integration
OGSAGlobus
COG
CHEFToolkit
Gridport
AlliancePortal
Workflow
Testing
NMIDist
Slide courtesy of Chuck Severance
OGCEOGCEConsortium
NMI Features List (April 04 Release)
• Content Management:– Resources: Users and groups can upload and share documents
and URLs. – News: Several news feeds are available, including XMLHack and
BBC News. – Grid Context: Store arbitrary web objects (movies, web pages,
audio files) in a customizable, annotated directory tree. • Collaborative:
– Chat: Engage other members of your group in online discussion. – Discussion: Participate in discussions with other members of your
group – Calendar Program:schedule appointments and attach associated
documents and URLs. Calendars can be used to schedule both user and group activities.
– Anabas Impromptu: system for real time shared display, audio, and chats.
OGCEOGCEConsortium
NMI Features List (April 04 Release)
• GT 3.0/Interactive:– Proxy Manager and Job Submit: Obtain and manage one or more Grid credentials
that can be used to access Grid resources through your browser. – GridFTP: Use Grid credential to browse remote directories and to upload and
download files between your desktop and a remote host. – Application Manager– Application Factory Manager
• Remote Job Management – GridAnt for workflow– Job submission portal– Community Scheduling Framework (CSF): submit jobs to scheduler
• Information Services:– LDAP Browser:– Navigate the LDAP server of your choice. – GridPort Information Repository Portlets (GPIR): Portlets are available for browsing
Machine Summaries, System Status, and Total Grid Capabilities.• Application Portlets:
– Specialized portlets for projects
OGCEOGCEConsortium
Key Feature – Download a Portal & Install
• Prerequisite Software– Java 2 Standard Edition version
1.4.x or higher. – Apache Ant version 1.5.x or higher.
• Preparing the Installation– Optionallly, create a "portal"
account on your web server. – Download the portal bundle – Put the download bundle into the
home directory of "portal" account. – Untar or unzip the portal bundle. – Globus: installation process will
create the directories .globus and .globus/certificates in the
• Software automatically installed includes:
– Globus/Java CoG– CHEF
• Install the Portal– Move into the portal/chef-1.0.7
directory.– Edit file "build.properites" – Type "ant deploy" to build in and
install the portal. – Start the Tomcat web server. – Point your browser at http://<your
host name>:10081/ogce and create an account.
• OS/Machines Tested:– MAC OSX– PC/Windows XP– Linux – Cygwin
OGCEOGCEConsortium
Current Status
• SC03 Demonstration portal and software download available– Beta1
• Updated Release available now– Bug fixes– Now supporting GT3.x with J2SDK 1.4.2
• Working on NMI April 2004 release– Including new features to support science portal applications
• Open GCE Runtime Engine (OGRE) workflow services and portlets• Sample template portlets for simple science applications
– Based on aggregation of existing projects– OGSA/OGSI/GT3.x compatible
OGCEOGCEConsortium
Future Directions
• Moving towards new portlet standard– JSR 168– Should allow compatibility with
• GridSphere• Academic education and business portals Oncourse, OneStart, CHEF
(of course), and uPortal • Major commercial products offerings from Sun, IBM, Oracle, etc.
• Incorporate new features:– Grid Services (CSF, OGSA-DAI portlets, Job Sequencer)– Access Grid, Polycom and other AV Portlets
• Collaborations and Contributions– DOE SciDAC projects (such as CMCS)
• Expand online repository– Develop process for third party contributions
OGCEOGCEConsortium
Future Directions
• Moving towards new portlet standard– JSR 168– Should allow compatibility with
• GridSphere• Academic education and business portals Oncourse, OneStart, CHEF
(of course), and uPortal • Major commercial products offerings from Sun, IBM, Oracle, etc.
• Incorporate new features:– Grid Services (CSF, OGSA-DAI portlets, Job Sequencer)– Access Grid, Polycom and other AV Portlets
• Collaborations and Contributions– DOE SciDAC projects (such as CMCS)
• Expand online repository– Develop process for third party contributions
OGCEOGCEConsortium
Participants
Indiana University
University of Michigan
University of Texas at Austin
Argonne National Labs
JavaCoG
National Center for Supercomputing Applications
the globus alliance