107
National Center for Supercomputing Applications Production Cyberenvironment for a A Computational Chemistry Grid PRAGMA13, NCSA 26 Sep 07 Sudhakar Pamidighantam NCSA, University of Illinois at Urbana-Champaign [email protected]

Production Cyberenvironment for a A Computational Chemistry Grid PRAGMA13, NCSA 26 Sep 07

  • Upload
    lester

  • View
    39

  • Download
    0

Embed Size (px)

DESCRIPTION

Production Cyberenvironment for a A Computational Chemistry Grid PRAGMA13, NCSA 26 Sep 07. Sudhakar Pamidighantam NCSA, University of Illinois at Urbana-Champaign [email protected]. Acknowledgements. Outline. Historical Background Grid Computational Chemistry - PowerPoint PPT Presentation

Citation preview

Page 1: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Production Cyberenvironment for a

A Computational Chemistry Grid

PRAGMA13, NCSA

26 Sep 07

Sudhakar PamidighantamNCSA, University of Illinois at

[email protected]

Page 2: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Acknowledgements

Page 3: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Outline

• Historical Background Grid Computational Chemistry

• Production Environments• Current Status Web Services • Usage (Grid and Science

Achievements)• Brief Demo • Future

Page 4: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

MotivationSoftware - Reasonably Mature and easy to use to address

chemists questions of interest

Community of Users - Need and capable of using the software Some are non traditional computational chemists

Resources - Various in capacity and capability

Page 5: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Background

Qauntum Chemistry Remote Job Monitor( Quantum Chemistry Workbench)1998, NCSA

Chemviz1999-2001, NSF (USA)http://chemviz.ncsa.uiuc.edu

TechnologiesWeb Based Client Server ModelsVisual InterfacesDistributed computing (Condor)

Page 6: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem

NCSA Alliance was commissioned 1998

Diverse HPC systems deployedboth at NCSA and Alliance Partner Sites

Batch schedulers different at sitesPolicies favored different classes and modes ofuse at different sites/HPC systems

Page 7: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Extended TeraGrid Facility

www.teragrid.org

Page 8: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

NSF Petascale Road Map• Track I Scheme Multi petaflop single site system to be

deployed by 2010 Several Consortia Competing (Now under review)

• Track 2 Sub petaflop systems Several to be deployed until Track 1 is online

First one will be at TACC ( 450 TFlops) Available Fall 2007( 50 000 Processors/Cores)

NCSA is deploying a 110 TFlops in April 2007(10000 Processors/cores)

Second subpetaflops systems being reviewed

Page 9: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Grid and GridlockAlliance lead to Physical Grid

Grid lead to TeraGrid

Homogenous Grid with predefined fixed software and system stack was planned (Teragrid) but it was difficult to keep it homogenous

Local preferences and diversity leads to heterogeneous grids now! (Operating Systems, Schedulers, Policies, Software and Services)

Openness and standards that lead interoperability are critical for successful services

Page 10: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Current Grid Status

Grid Hardware

Middleware

Scientific Applications

InterfacesInterfaces

Page 11: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

User Community

Chemistry and Computational BiologyUser BaseSep 03 – Oct 04

NRAC AAB Small Allocations------------------------------------------------------------- #PIs 26 23 64

#SUs 5,953,100 1,374,100 640,000

Page 12: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Page 13: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Some User Issues Addressed by the new Services

• New systems meant learning new commands• Porting Codes• Learning new job submissions and monitoring

protocols• New proposals for time (time for new proposals)• Computational modeling became more popular

and number of users increased (User Management)

• Batch queues are longer / waiting increased• Finding resources where to compute

complicated - probably multiple distributed sites

• Multiple proposals/allocations/logins• Authentication and Data Security • Data management

Page 14: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Computational Chemistry Grid

This is a Virtual Organization

Integrated Cyber Infrastructure for Computational Chemistry

Integrates Applications, Middleware, HPC resources, Scheduling and Data management

Allocations, User services and Training

Page 15: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Resources

System (Site) Procs Avail

Total CPU Hours/Year

Status

Intel Cluster (OSC) 36 315,000 SMP and Cluster nodes

HP Integrity Superdome (UKy)

33 290,000 TB Replaced with an SMP/ Cluster nodes

IA32 Linux Cluster (NCSA)

64 560,000 Allocated

Intel Cluster (LSU) 1024 1,000,000 Allocated

IBM Power4 (TACC) 16 140,000 AllocatedTeragrid (Multiple

Institutions) 2-10000 250,000 New Allocations

Expected

The initial Acesss Grid Testbed Nodes (38) and Condor SGI resources (NCSA, 512 nodes) have been retired this year.

Page 16: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Other Resources

Extant HPC resources at variousSupercomputer Centers (Interoperable)

Optionally Other Grids and Hubs/local/personal resources

These may require existingallocations/Authorization

Page 17: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Page 18: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Grid Middleware Proxy Server

GridChem System

user user useruser user

PPortal Clientortal Client

Grid ServicesGrid Services

GridGrid

applicationapplicationapplicationapplication

Mass Storage

http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0438312

Page 19: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Applications• GridChem supports some apps already

– Gaussian, GAMESS, NWChem, Molpro, ADF,QMCPack, Amber

• Schedule of integration of additional software– ACES-3– Crystal– Q-Chem– Wein2K– MCCCS Towhee – Others...

Page 20: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem User ServicesAllocation Request

https://www.gridchem.org/allocations/comm_form.php

Page 21: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem User ServicesConsulting Ticketing System

User View

Page 22: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem User ServicesConsulting Ticketing System

https://www.gridchem.org/consult/Consultants View

Page 23: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Gridchem Middleware Service (GMS)

Page 24: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GrdiChem Web ServicesQuick Primer

XML is used to tag the data, SOAP is used to transfer the data, WSDL is used for describing the services available and UDDI is used for listing what services are available.

Web Services is different from Web Page Systems or Web Servers:There is no GUI

Web Services Share business logic, data & processes through APIs with each other (not with user)

Web Services describe Standard way of interacting with “web based” applications

A client program connecting to a web service can read the WSDL to determine what functions are available on the server. Any special datatypes used are embedded in the WSDL file in the form of XML Schema. Universal Description, Discovery, and Integration. WSRF Standards Compliant.

Page 25: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Web Services Client Objects Database Interaction

WSResources

DTO

Objects Hibernate

Databasehb.xml

Client

DTO (Data Transfer Object)Serialize transfer through XML

DAO (Data Access Object) How to get the DB objectshb.xml (Hibernate Data Map)

describes obj/column data mapping

BusinessModel

DAO

Page 26: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Data Models

Users Projects Resources

UserProjectResource

SoftwareResources

ComputeResources

NetworkResources

StorageResources

ResourcesresoruceIDTypehostNameIPAddresssiteID

userIDprojectIDresourceIDloginNameSUsLocalUserUsed

JobsjobIDjobNameuserIDprojIDsoftIDcost

Users Resources

Page 27: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Computational Chemistry Resource

Page 28: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GMS_WS Use Cases

• Authentication• Job Submission• Resource Monitoring• Job Monitoring• File Retrieval• …

http://www.gridchem.org:8668/space/GMS/usecase

Page 29: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

• GetResourceProperty• SetTerminationTime • Destroy• Create • Login • LoadVO • RetrieveFiles • LoadFiles • DeleteFiles • LoadParentFiles • RefreshFiles • MakeDirectory • SubmitJob • SubmitMultipleJobs • PredictJobStartTime • KillJob • HideJob • UnhideJob • UnhideJobs • DeleteJob • FindJobs • GetJobStatus • RetrieveJobOutput • RetrieveNextDataBlock • StopFileAction • GetUserPreferences • PutUserPreferences

GridChem Web Services Operations

Page 30: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GMS_WS Authentication

• WSDL (Web Service Definition Language) is a language for describing how to interface with XML-based services. It describes network services as a pair of endpoints operating on messages with either document-oriented or procedure-oriented information.

• The service interface is called the port type • WSDL FILE: <?xml version="1.0" encoding="UTF-8"?> <definitions name=“GMS"

targetNamespace=http://www.gridchem.org/gms " xmlns="http://schemas.xmlsoap.org/wsdl/" …

http://www.gridchem.org:8668/space/GMS/usecase

Contact GMSCreates Session, Session RP and EPRSends EPR ( Like a Cookie, but more than that)

Login Request(username:passwd)

Validates, Loads UserProjectsSends acknowledgement

Retrieve UserProjects(GetResourceProperty Port Type [PT])

GridChem Client GMS

Page 31: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GMS_WS Authenticationhttp://www.gridchem.org:8668/space/GMS/usecase

Selects projectLoadVO port type(w. MAC address)

Verifies user/project/MACaddrLoad UserResources RP

Retrieve UserResources[as userVO/ Profile](GetResourceProperty port Type PT)

GridChem Client GMS

Validates, Loads UserProjectsSends acknowledgement

Sends acknowledgement

Page 32: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GMS_WS Job Submission

Create Job objectPredictJobStartTime PT + JobDTO

JobStart Prediction RP

PT = portType RP = Resource PropertiesDTO = Data Transfer Object

Completion:Email from batch systemto GMS servercron@GMS DB

SubmissionCoGKitGAT“gsi-ssh”

If decision OK,SubmitJob PT + JobDTO

Create Job objectAPI—SubmitStore Job Object

Send Acknowledgement

Need to check to make sure allocation-time is available.

GC Client GMS

Page 33: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GMS_WS Monitoring

Parse XML,Display

PT = portType RP = Resource PropertiesDTO = Data Transfer ObjectDB = Data Base

cron@GMS servercron@HPC ServersJob Launcher NotificationsVO Admin emailparses email DB(status + cost)

Request for Job,Resource StatusAlloc. Balance

UserResource RP Updated from DB

GC Client GMS Resources/Kits/DB

Send info

Discover Applications (Software Resources)

Monitor System

Monitor Queues

Page 34: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GMS_WS Job Status

Job Status jobDTO.status Job Launcher

Status Update

Estimate Start time

Scheduler emails/

notificationsNotifications: Client, email, IM

GC Client GMS Resources/Kits/DB

Page 35: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GMS_WS File Retrieval (MSS)

GetResourceProperty PTFileDTO(?)LoadFile PT(project folder+job)

Validates projectfolder owned by user.Send new listing

PT = portType RP = Resource PropertiesDTO = Data Transfer ObjectMSS = Mass Storage System

Job Completion:Send Output to MSS

LoadFile PT MSS queryUserFiles RP +FileDTO object

Retrieve Root Dir. Listing on MSS withCoGKit orGAT or“gsi-ssh”

API file requestStore locallyCreate FileDTOLoad into UserData RP

RetrieveFiles PT(+file rel.path)

Retrieve file:CoGKit orGAT or“gsi-ssh”

GetResourceProperty PT

GC Client GMS Resources/Kits/DB

Page 36: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GMS_WS File Retrieval

PT = portType RP = Resource PropertiesDTO = Data Transfer ObjectMSS = Mass Storage System

Create FileDTO (?)Load into UserData RP

RetrieveJobOutput PT(+JobDTO) Job Record from

DB.Running: from ResourceComplete: from MSS

Retrieve file:CoGKit orGAT or“gsiftp”

GetResourceProperty PT

GC Client GMS Resources/Kits/DB

Page 37: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Web ServicesWSRF (Web Services Resource Framework) Compliant

WSRF Specifications:WS-ResourceProperties (WSRF-RP) WS-ResourceLifetime (WSRF-RL) WS-ServiceGroup (WSRF-SG) WS-BaseFaults (WSRF-BF)

%ps -aux | grep ws/usr/java/jdk1.5.0_05/bin/java \-Dlog4j.configuration=container-log4j.properties \-DGLOBUS_LOCATION=/usr/local/globus \-Djava.endorsed.dirs=/usr/local/globus/endorsed \-DGLOBUS_HOSTNAME=derrick.tacc.utexas.edu \-DGLOBUS_TCP_PORT_RANGE=62500,64500 \-Djava.security.egd=/dev/urandom \-classpath /usr/local/globus/lib/bootstrap.jar: /usr/local/globus/lib/cog-url.jar: /usr/local/globus/lib/axis-url.jar org.globus.bootstrap.Bootstrap org.globus.wsrf.container.ServiceContainer -nosec

Logging ConfigurationWhere to find Globus

Where to get random seedfor encryption key generation

Classpath (required jars)

Page 38: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Software OrganizationOpen Source Distribution

• CVS for GridChem

Page 39: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

• Package:org.gridchem.service.gms

GMS_WS

Page 40: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GMS_WS

+

Should these each be a separate package?

Page 41: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

model

dto

credential

jobnotification

file file.taskjob.task

user

exceptions

resource

persistence

synchquery

test

util

dao

gpir

cryptenumeratorsgatproxy

GMS_WS

client

audit

gms Classes for WSRF service implementation (PT)Cmd line tests to mimic client requestsData Access Obj – queries DB via persistent classes (hibernate)Data Transfer Obj – (job,File,Hardware,Software,User) XMLHow to handle errors (exceptions)CCG Service business mode (how to interact)Contains user’s credentials for job sub. file browsing,…“Oversees correct” handling of user data (get/putfile).Define Job & util & enumerations (SubmitTask, KillTask,…)

CCGResource&Util, Synched by GPIR, abstract classesNetworkRes., ComputeRes., SoftwareRes., StorageRes., VisualizationRes.User (has attributes – Preference/Address)DB operations (CRUD), OR Maps, pool mgmt,DB session,Classes that communicate with other web servicesPeriodically update DB with GPIR info (GPIR calls)JUnit service test (gms.properties): authen. VO retrieval, Res.Query,Synch, Job Mgmt, File Mgmt, NotificationContains utility and singleton classes for the service.Encryption of login passwordMapping from GMS_WS enumeration classes DBGAT util classes: GATContext & GAT Preferences generationClasses deal with CoGKit configuration.

Autonomous notification via email, IM, textmesg.

Page 42: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GMS_WS external jars

• Testing• For XML Parsing• “Java” Document Object Model

– Lightweight– Reading/Writing XML Docs– Complements SAX (parser) & DOM– Uses Collections**

Page 43: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Resources Monitoring

http://portal.gridchem.org:8080/gridsphere/gridsphere?cid=home

Page 44: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Resources

New Computing Systems System Capacity (Cpus/Cores) Capability

Mercury(NCSA) 1774 Small/Large Parallel Runs

Abe(NCSA) 9600 Massively Parallel Runs

DataStar(SDSC) 2368 SharedMemory Large Runs

Bluegene/L(SDSC) 3456 Cluster Large Parallel Runs

TeragridCluster(SDSC) 564 Small/Large Parallel Runs

BigRed(IU) 1024 SharedMemory Small/Large Runs

BCX (UKy) 1360 Shared/Distributed Memory small/Large Parallel Runs

Page 45: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Application Software ResourcesCurrently Supported

Suite Version Location

Gaussian 03 C.02/D.01 Many Platforms

MolPro 2006.1 NCSA

NWChem 5.0/4.7 Many Platforms

Gamess Jan 06 Many Platforms

Amber 8.0 Many Paltforms

QMCPack 2.0 NCSA

Page 46: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Software ResourcesNew Applications

Integration Underway

• ADF Amsterdam Density Functional Theory• Wien2K Linearized Augemented Plain wave (DFT)• CPMD Car Parinello Molecular Dynamics • QChem Molecular Energetics (Quantum Chemistry)• Aces3 Parallel Coupled Cluster Quantum Chemistry• Gromacs Nano/Bio Simulations (Molecular Dynamics)

• NAMD Molecular Dynamics• DMol3 Periodic Molecular Systems ( Quantum Chemistry)• Castep Quantum Chemistry • MCCCS-Towhee Molecular Confirmation Sampling (Monte Carlo)• Crystal98/06 Crystal Optimizations (Quantum Chemistry)• ….

Page 47: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem User Services• Allocationhttps://www.gridchem.org/allocations/index.shtmlCommunity and External Registration Reviews, PI Registration and Access Creation Community User Norms Established

• Consulting/User Serviceshttps://www.gridchem.org/consultTicket tracking, Allocation Management

• Documentation, Training and Outreachhttps://www.gridchem.org/doc_train/index.shtmlFAQ Extraction, Tutorials, Dissemination

Help is integrated into the GridChem client

Page 48: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Users and Usage

• 242 Users under 128 ProjectsInclude Academic PIs, two graduate

classesAnd about 15 training usersMore than a 442000 CPU Wallhours

since Jan 06More than 10000 Jobs processed

Page 49: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Science Enabled

• Azide Reactions for Controlling Clean Silicon Surface Chemistry: Benzylazide on Si(100)-2 x 1Semyon Bocharov et al..J. Am. Chem. Soc., 128 (29), 9300 -9301, 2006

• Chemistry of Diffusion Barrier Film Formation: Adsorption and Dissociation of Tetrakis(dimethylamino)titanium on Si(100)-2 × 1 Rodriguez-Reyes, J. C. F.; Teplyakov, A. V.J. Phys. Chem. C.; 2007; 111(12); 4800-4808.

• Computational Studies of [2+2] and [4+2] Pericyclic Reactions between Phosphinoboranes and Alkenes. Steric and Electronic Effects in Identifying a Reactive Phosphinoborane that Should Avoid Dimerization Thomas M. Gilbert* and Steven M. Bachrach Organometallics, 26 (10), 2672 -2678, 2007.

Page 50: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Science Enabled• Chemical Reactivity of the Biradicaloid (HO...ONO) Singlet

States of Peroxynitrous Acid. The Oxidation of Hydrocarbons, Sulfides, and Selenides. Bach, R. D et al. J. Am. Chem. Soc. 2005, 127, 3140-3155.

• The "Somersault" Mechanism for the P-450 Hydroxylation of Hydrocarbons. The Intervention of Transient Inverted Metastable Hydroperoxides. Bach, R. D.; Dmitrenko, O. J. Am. Chem. Soc. 2006, 128(5), 1474-1488.

• The Effect of Carbonyl Substitution on the Strain Energy of Small Ring Compounds and their Six-member Ring Reference Compounds Bach, R. D.; Dmitrenko, O. J. Am. Chem. Soc. 2006,128(14), 4598.

Page 51: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Client Download Statistics

http://download.gridchem.org/usage/

Page 52: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Distribution of GridChem User Community

Page 53: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Job Distribution Job Distribution by Time

0

50

100

150

200

250

300

350

100

300

500

700

900

1100

1300

1500

1700

1900

2100

2300

2500

2700

2900

Wall Clock x CPUs (~SUs)

Num

ber o

f Job

s

Job Distribution by Time

0

20

40

60

80

100

120

140

2 4 6 8

10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50

Wall Clock Time x CPUs (~SUs)Nu

mbe

r of J

obs

Page 54: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

System Wide UsageHPC System Usage (SUs)

Tungsten(NCSA) 5507Copper(NCSA) 86484CCGcluster(NCSA) 55709Condor(NCSA) 30SDX(UKy) 116143CCGCluster(UKy) .5Longhorn(TACC) 54CCGCluster(OSC) 62000TGCluster(OSC) 36936Cobalt(NCSA) 2485Champion(TACC) 11Mike4 (LSU) 14537

Page 55: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Client Enhancements

• New Molecular Editor JMolEditor (ANU) Integration• VMD Is integrated• Nanotube Generator (Tubegen) Will be

available• Gamess Graphical User Interphase

Page 56: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Java Molecular Editor• JMolEditorThree Dimensional Visual with Java 3D

Intuitive Molecule ManipulationInteractive Bond, Angle and Dihedral Settings

A Gaussian input generator Interface

Page 57: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Nanotube Generator:Tubegen

Courtesy: Doren Research Group at the University of Delaware Crystal Cell Types

Output Formats

Page 58: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Gamess GUI

Page 59: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Post Processing• IR/Raman Spectra now accessible from G03, MolPro,

NWChem and Gamess Suites VCD/ROA To be Included

Page 60: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Post Processing

• Normal Mode Viewing in 3D VRML• Other Spectra With MO Integration NMR Electronic Spectra

Page 61: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem UsabilityDynamic Information

Page 62: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem Usability

• Information on Potential Start and End Time for a given set of Job parameters

• Automated Resource Selection • Possible Job Migration In case of

dropped nodes or incomplete job• Monitoring Multiple Jobs • Automated Monitoring Job Output

Page 63: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

• Implementation of GRMS resource management Service http://www.gridlab.org/WorkPackages/wp-9

• Moving toward Service based job submission eliminating gateway interfaces

• Infrastructure for multiple input files for single application

• Infrastructure for multiple inputs in High Throughput processing

• Integrated workflow for multi scale coupled modeling

• Meta-scheduling for High Throughput Processing Match Making, Round-robin scheduling, Preferred Host Set usage

GridChem Middleware InfrastructureImplementation Currently underway

Page 64: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

GridChem In New CollaborationsResource Providers

• New Resource Providers Open Science Grid Initially for Bio-related applications (open

source preferably)

• PRAGMA Partner sites University of Hyderabad

• ORNL (Could be via TeraGrid)

• International Partners KISTI, APAC, Daresbury Labs

Page 65: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Scientific Collaborations

• GridChem Extension to Molecular Sciences (Bio, Nano, Geo and Materials Sciences) (NSF Proposal)

• Parameter Sweep for Potential Energy Hyper Surfaces (Faculty Fellows, NCSA)

• Automated Parameterization of Force fields (NSF Proposal)

• Ab initio Molecular Dynamics (Faculty Fellows, NCSA)

• Education (CI-TEAM) (NSF Proposals)

• Multi-Scale Modeling (IACAT, UIUC)

Page 66: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Some New GridChem Infrastructure• Workflow Editors• Coupled Application Execution• Large Scale Computing• Metadata and Archiving • Rich Client Platform Refactorization• Intergrid Interactions

• Open Source Distribution http://cvs.gridchem.org/cvs/

• Open Architecture and Implementation details http://www.gridchem.org/wiki

Page 67: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Critical Gateways Issues

• Science Gateways compete with business as usual for the end user research scientist

• No direct access to HPC systems may be possible leading to apparent lack of control for users

• No “End to end solutions” If part of the research needs require old ways

Gateways may be avoided• Learning to use Gateways should provide substantial

added benefit –Cost/Benefit Issues for users• Flexibility to integrate new applications as needed by

community quickly is critical to keep the user community engaged

Page 68: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Authentication

Page 69: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Resource Status

Page 70: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Job Editor

Page 71: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Job Submission

Page 72: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Job Monitoring

Page 73: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Gradient Monitoring

Page 74: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Energy Monitoring

Page 75: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Post Processing

Page 76: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Visualization

Molecular Visualization

Electronic Properties

Spectra

Vibrational Modes

Page 77: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Molecular VisualizationBetter molecule representations(Ball and Stick/VDW/MS)

In Nanocad Molecular Editor Third party visualizer integration Chime/VMD

Export Possibilities to others interfaces Deliver standard file formats

(XML,SDF,MSF,Smiles etc…)

Page 78: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Eigen Function Visualization

• Molecular Orbital/Fragment Orbital• MO Density Visualization• MO Density Properties• Other functionsRadial distribution functions

Page 79: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Some example VisualsArginine Gamess/6-31G*Total electronic density

2D - Slices

Page 80: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Electron Density in 3DInteractive (VRML)

Page 81: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Orbital 2D DisplaysN2 6-31g* Gamess

Page 82: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Orbital 3DVRML

Page 83: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Spectra

• IR/Raman Vibrotational Spectra• UV Visible Spectra

• Spectra to Normal Modes• Spectra to Orbitals

Page 84: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Possible H-bonds network for P450cam

hydroperoxy intermediate

C

O

HO

H

OO H

Fe3+

OH

CH3

HO

H

H

N

GLY248 peptide

VAL253 peptide

THR252

2.98Å2.79Å

2.99Å

2.75Å

3.16Å

3.07Å

3.32Å

Suggested:

THR252 accepts an H-bond from the hydroperoxy (Fe(III)-OOH that promotes thesecond protonation on the distal oxygen, leading to the O-O bond cleavage

Nagano, S.; Poulos, T.L. J. Biol. Chem. 2005, 250, p.1668• Auclair, K.; Hu, Z.; Little, D. M.; Ortiz de Montellano, P. R.; Groves, J. T. J. Am.

Chem. Soc. 2002, 124, 6020.

Page 85: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

The Somersault Isomerization of Model Cpd0

TS

Fe

OO H

SHFe

O

SH

H

O

E24.8 (24.3) 20.3 kcal/mol

1.447Å

2.226Å

2.408Å

1.869Å116.7

102.0

97.9

1.665Å

2.473Å

1.662Å

97.2

177.9

E = 17.5 (17.8) kcal/mol

EH-bonding = 17.0 kcal/mol

77.0

127.4

97.8

2.437Å

vi=101.5i cm-1

GS MIN

CCCC CC N

C CC

S

C

O

NFeNC

O

CC CN CC CC

CC

CCC CC

NC

CC

N

S

CC Fe

O

CC N

O

C CC N CC CCC

CC CCCCC N

CC N

S

2.487Å

CFe

1.658Å

C

O

2.186Å

O

N CC N CCC CC CC

vi=93.7i cm-1

Robert Bach and Olga Dmytrenko, 2006

Page 86: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Energy Diagram for the Concerted Non-synchronous Hydroxylation of Isobutane

Fe

SH

O

O(H3C)3C

H

H

Fe

SH

O

OH

H(H3C)3C

Fe

SH

O

OH

H(H3C)3C

MIN-26a

Fe

SH

O

O HH

(H3C)3C

SH

O

OH

H

(H3C)3C

Fe

Fe

SH

OH

OH

(H3C)3C

-4.0

-19.2

17.2

-83.7

MIN-24bTS-25

MIN-26b

PRODUCT 28

3.848Å5.5

11.7

TS-27

GS-24a

19.5

Fe

SH

O

(H3C)3COHH

Energy diagram (kcal/mol) for the oxidation of the isobutane with ground state, 24a (GS-8 hydrogen bonded to isobutane). MIN-24b [model oxidant MIN-10 (PorFe(SH)OHO) hydrogen bonded to isobutene] is not necessarily on the reaction pathway.

Page 87: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Somersault Mechanism Summary for Isobutane Hydroxylation

S

FeIV

O

CH

CH3

CH3H3C

HO

S

FeIV

O

C

HCH3

CH3H3C

O

H

S

FeIV

O

C

H CH3

CH3H3C

O

H

S

FeIV

O

C

H

CH3

CH3H3C

O

H

Page 88: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

TetrakisDimethylAminoTitanium and its derivatives on Si(100)-2x1 Surface: Diffusion Barrier Thinfilms on Silicon

Rodrigues-Reyes and Teplyakov

Page 89: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Benzylazide on Si(100)-2x1 SurfaceDeposition of Aromatic Moieties on Silicon for Lateral Electron

TransferBocharov et al..

Page 90: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

[2+2] Cyclo Additions involving B=P BondsGilbert and Bachrach

Dimerization

Ethyne Addition

Ethene Additions

Page 91: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Possible H-bonds network for P450cam

hydroperoxy intermediate

C

O

HO

H

OO H

Fe3+

OH

CH3

HO

H

H

N

GLY248 peptide

VAL253 peptide

THR252

2.98Å2.79Å

2.99Å

2.75Å

3.16Å

3.07Å

3.32Å

Suggested:

THR252 accepts an H-bond from the hydroperoxy (Fe(III)-OOH that promotes thesecond protonation on the distal oxygen, leading to the O-O bond cleavage

Nagano, S.; Poulos, T.L. J. Biol. Chem. 2005, 250, p.1668• Auclair, K.; Hu, Z.; Little, D. M.; Ortiz de Montellano, P. R.; Groves, J. T. J. Am.

Chem. Soc. 2002, 124, 6020.

Page 92: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

The Somersault Isomerization of Model Cpd0

TS

Fe

OO H

SHFe

O

SH

H

O

E24.8 (24.3) 20.3 kcal/mol

1.447Å

2.226Å

2.408Å

1.869Å116.7

102.0

97.9

1.665Å

2.473Å

1.662Å

97.2

177.9

E = 17.5 (17.8) kcal/mol

EH-bonding = 17.0 kcal/mol

77.0

127.4

97.8

2.437Å

vi=101.5i cm-1

GS MIN

CCCC CC N

C CC

S

C

O

NFeNC

O

CC CN CC CC

CC

CCC CC

NC

CC

N

S

CC Fe

O

CC N

O

C CC N CC CCC

CC CCCCC N

CC N

S

2.487Å

CFe

1.658Å

C

O

2.186Å

O

N CC N CCC CC CC

vi=93.7i cm-1

Robert Bach and Olga Dmytrenko, 2006

Page 93: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Energy Diagram for the Concerted Non-synchronous

Hydroxylation of Isobutane

Fe

SH

O

O(H3C)3C

H

H

Fe

SH

O

OH

H(H3C)3C

Fe

SH

O

OH

H(H3C)3C

MIN-26a

Fe

SH

O

O HH

(H3C)3C

SH

O

OH

H

(H3C)3C

Fe

Fe

SH

OH

OH

(H3C)3C

-4.0

-19.2

17.2

-83.7

MIN-24bTS-25

MIN-26b

PRODUCT 28

3.848Å5.5

11.7

TS-27

GS-24a

19.5

Fe

SH

O

(H3C)3COHH

Energy diagram (kcal/mol) for the oxidation of the isobutane with ground state, 24a (GS-8 hydrogen bonded to isobutane). MIN-24b [model oxidant MIN-10 (PorFe(SH)OHO) hydrogen bonded to isobutene] is not necessarily on the reaction pathway.

Page 94: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Somersault Mechanism Summary for Isobutane Hydroxylation

S

FeIV

O

CH

CH3

CH3H3C

HO

S

FeIV

O

C

HCH3

CH3H3C

O

H

S

FeIV

O

C

H CH3

CH3H3C

O

H

S

FeIV

O

C

H

CH3

CH3H3C

O

H

Page 95: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Unsymmetrical Mo(CO)4 Crown Ethers

Page 96: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Dibenzaphosphepin based bis(phosphorous)polyether chelated

Mo(CO)4

Page 97: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Crystal Structures

CSD:XAPZAP

cis-(6,6'-((1,1'-Binaphthyl)-2,2'-diylbis(oxy))bis(dibenzo(d,f)(1,3,2)dioxaphosp hepin))-tetracarbonyl-molybdenum(0) C48 H28 Mo1 O10 P2

CSD:DEQDOS

cis-Tetracarbonyl-(P,P'-(6-(2'-oxy-2-biphenyl)-3,6-dioxa-hexanolato)-bis(dibenzo (d,f)(1,3,2)dioxaphosphepine)-P,P')-molybdenum C44 H32 Mo1 O12 P2

Page 98: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Starting Structure

Page 99: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Optimized Structure

Page 100: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Reference Structure for Comparison

8

7

Page 101: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Structural ComparisonsC-C Torsion Angles for the OCH2CH2O Fragments and for the Axially

Chiral Biaryl Groups

Atoms PCMODEL* UFF Ab Initio Amber

C37-C42-C43-C48 -49.9 -26.4 -43.0 -40.4C1-C6-C7-C12 45.4 22.3 -22.3 -72.8C13-C22-C23-C32 75.6 74.7 -85.9 -81.2C32-O-C33-C34 -178.4 -140.8 159.7 -171.2O-C33-C34-O 62.4 -64.5 -87.3 -82.4C33-C34-O-C35 -80.6 -118.9 67.8 64.9C34-O-C35-C36 174.6 118.9 -153.4 60.1O-C35-C36-0 66.2 56.0 64.0 67.3

• *Hariharasarma, et al. Organomet., 1232-1238, 2000.• Ab Initio=B3LYP/3-21G*• Amber9 ff03, GAFF, chloroform, 300K, median over 1ns MD

Page 102: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

MD OCH2CH2O Structure

8

7

Page 103: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

MD Biaryl Structure

Page 104: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

1H NMR Chemical Shift ComparisonFor Aromatic Protons

Reference 32ppm (from TMS B3LYP/6-31g*)Atom Exp. Abinitio Atom Exp. AbinitioH2 7.025 5.6 H25 6.578 5.7H3 7.026 5.8 H26 6.737 5.9H4 7.049 5.9 H27 7.018 6.1H5 7.181 6.0 H28 7.623 6.5

H8 7.110 6.1 H30 7.790 6.7H9 6.890 6.0 H31 7.289 6.9H10 6.721 6.0H11 6.237 5.7 H38 7.327 6.2

H39 7.274 6.1H14 7.925 5.8 H40 7.169 6.0H15 7.808 6.3 H41 7.350 6.3

H17 7.741 6.0 H44 7.360 6.1H18 7.254 5.6 H45 7.160 5.9H19 7.091 5.1 H46 7.176 6.0H20 6.989 4.6 H47 7.060 7.0

Page 105: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Third Year Plans• Post Processing Spectra and related entities• New Application SupportAces3, Dmol3, Vasp,…..• Expansion of ResourcesTeragrid, OSG, Pragma Systems and New

resources at Partner Sites• Extension PlanTwo Proposals in review for Extension

Page 106: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Future Plans

• Preparations for Petaflop computingHigh throughput massively parallel applications

• Complex workflows for integrating multiple interdependent applications

Multiscale Computing

• Archiving and annotating data for future useOpen Data initiatives by NIH and NSF

Page 107: Production Cyberenvironment for a  A Computational Chemistry Grid    PRAGMA13, NCSA   26 Sep 07

National Center for Supercomputing Applications

Acknowledgments

• Rion Dooley, TACC Middleware Infrastructure• Stelios Kyriacou, OSC Middleware Scripts• Chona Guiang, TACC Databases and Applications

• Kent Milfeld, TACC Database Integration • Kailash Kotwani, NCSA, Applications and Middleware

• Scott Brozell, OSC, Applications and Testing• Michael Sheetz, UKy, Application Interfaces• Vikram Gazula, UKy, Server Administration• Tom Roney, NCSA, Server and Database Maintenance