Louisiana State Grid

Dick Greenwood

Developments in the Louisiana State Grid (LONI)

Dick Greenwood

Louisiana Tech University

DOSAR III Workshop at

The University of Oklahoma

September 21-22, 2006

21SEP06 2Louisiana State Grid

Dick Greenwood

LONI Background

• In September 2004, the Louisiana State has committed $40M for a state-wide optical network.

• 40Gb/sec bandwith• Spanning 6 Universities

and 2 Health Centers:– LSU– Latech– UL-Lafayette– Tulane– UNO– Southern University– LSU Health Centers in

• New Orleans• Shereveport

21SEP06 3Louisiana State Grid

Dick Greenwood

2001 Plan

Louisiana State Grid

Dick Greenwood

Not an Ordinary Optical Network

21SEP06 5Louisiana State Grid

Dick Greenwood

Systems DeployedIBM p5 Systems– 1 February 2006 – LaTech [bluedawg] AIX v5.3● 17 users submitted 307 jobs– 15 March 2006 – Tulane [ducky] AIX v5.2● 43 users submitted 2946 jobs– 7 August 2006 – ULL [zeke] AIX v5.3● 9 users submitted 286 jobs– 8 September 2006 – UNO [neptune] AIX v5.3● 4 users submitted 10 jobs– ~October 2006 – SUBR [lacumba] AIX v5.3● 89 total LONI users

21SEP06 6Louisiana State Grid

Dick Greenwood

LONI Software Stack

• 100% TeraGrid compatible • …• Globus Toolkit• Condor• Virtual Data Toolkit (VDT)• … • The complete SW stack still to

be finalized

21SEP06 7Louisiana State Grid

Dick Greenwood

New Dell Linux Clusters To Be Delivered

• “Remote” Systems– 132 nodes, four-way Intel “Woodcrest”2.33 GHz, 4 GB RAM [4.921 TF; 528 GB RAM]– Shipment expected week of 25 September– First deployment will be LSU's system

• Next deployment likely LaTech

• Central System– 720 nodes, eight-way Intel “Cloverto(w)n”2.33 GHz,4 GB RAM [53.683 TF; 2.88 TB RAM]– Shipment expected end of November or earlyDecember

21SEP06 8Louisiana State Grid

Dick Greenwood

Linux ClustersWoodcrest and Cloverto(w)n– Woodcrest is dual-core on single silicon with ashared 4 MB cache– Cloverto(w)n is two dual-core modules in a

singlemodule (may change to four cores on singlesilicon)– Both based on 65 nm technology initially then45 nm– Both are capable of retiring four floating pointoperations per clock cycle, rather than two [IBM'sPOWER5 also does four floats per cycle]

21SEP06 9Louisiana State Grid

Dick Greenwood

Dell Cluster DescriptionEnvironmentals:– 208 V / 310 amperes; 64.5 KW; 18 tons of

cooling(7,868 cfm)– 6 racks total (4 node,1 control, 1 storage)– Rack Dimensions:78.7"Hx23.94"Wx39.93"D– Each rack has 4 PDUs;4 L6-30 208v connects(total of 24 L6-30circuits)

21SEP06 10Louisiana State Grid

Dick Greenwood

StorageCurrently very tight:– /home is 25 GB and /scratch is 280 GB– this limits usability– all served via Network File System (NFS) which isnot high performance● Future:– When central Linux cluster comes, it will include:● 14.0 TB raw at each “remote” site in one rack● 225 TB raw at the central site– Will provide central /home storage as well as global/scratch space– Using Lustre filesystem supported by ClusterFilesystems, Inc.

21SEP06 11Louisiana State Grid

Dick Greenwood

Award of NSF MRI: PetaShare1 x Overland Storage NEO Series 8000 tape storage system – 400 TB

Capacity: $ 153,558Specifications : Model 500, 12 Native Fibre LTO-3 Tape Drives, 500

Tape Slots, Redundant Power, Remote Access & Control, Touch Screen Front Panel (Quote includes

shipping and assembly)Deployment site: Louisiana State University5 x Overland Storage REO Series 9000 disk storage system - 44 TB

capacity each: 5 x $96,144 = $480,720Specifications : 44TB Raw SATA, 38 TB usable RAID 5, Protecton

OS, Remote Access and Control, Rackmount (Quote includes shipping and assembly)Deployment sites: Louisiana State University, Louisiana Tech

University, Tulane University, University of Louisiana at Lafayette, Tulane University, University of New

Orleans,Total requested equipment cost: $634,278

21SEP06 12Louisiana State Grid

Dick Greenwood

Tevfik’s Schedule for PetaShareYear 1: The required equipment for the development of PetaShare will be purchased,

installed at each participatingsite, calibrated, and will be integrated with the existing equipment at these sites. In

parallel to this process,we will start developing the key technologies necessary for this instrumentation: data-

aware storage, data-awareschedulers, and remote data analysis and visualization.

 Year 2: Technologies developed during the first year will be integrated. Transparent

storage selection, dataarchival, cataloging and remote access techniques will be developed on top of this

integrated system. A userfriendly and uniform interface will be developed for interaction.

 Year 3: The first prototypes will be deployed at the participating sites. Testing and

further development will beperformed. Application groups will start using the new instrumentation actively.

 Year 4: The developed system will be ported to other platforms. It will be made publicly available to the

community. Commercialization potential will be investigated.

We are currently in the process of negotiating with different vendors. The purchase of the equipment may happen

sometime in late Decemebr or early January.

21SEP06 13Louisiana State Grid

Dick Greenwood

THEEND

21SEP06 14Louisiana State Grid

Dick Greenwood

EXTRA SLIDES

21SEP06 15Louisiana State Grid

Dick Greenwood

21SEP06 16Louisiana State Grid

Dick Greenwood

Coastal Modeling• Hurricane Track

Prediction• Storm Surge

Modeling• Predicting Wind

Effects• Coastal Erosion

Modeling• Emergency

Response

21SEP06 17Louisiana State Grid

Dick Greenwood

SCOOP Project• SCOOP Portal

17

21SEP06 18Louisiana State Grid

Dick Greenwood

Other Applications• Numerical Relativity• Petroleum Engineering• Computational Fluid Dynamics (CFD)• High Energy Physics• Bioinformatics

21SEP06 19Louisiana State Grid

Dick Greenwood

RamahRamahLONI DWDM

LandryLandryLONI DWDM

AlexandriaAlexandriaLONI POP

ULLULLLONI POPOTM POP

CoushattaCoushattaLONI DWDM

DerryDerryLONI DWDM

55Km55Km

80Km80Km

89Km89Km

LSULSU

HammondHammondLONI POP

UNOUNOLONI POP

TulaneTulaneLONI POP

LSU HSC NOLSU HSC NOLONI POPOTM POP

86Km86Km 48Km48Km

54Km54Km

75Km75Km 20Km20Km

70Km70KmSouthernSouthern

DWDM LoopDWDM Loop

MonroeMonroeLONI POP

La TechLa TechLONI POP

82Km82Km

55Km55Km

82Km82Km

72Km72Km

61Km61Km

82Km82Km

Northern DWDM LoopNorthern DWDM Loop

SouthernSouthernLONI POP

MindenMindenLONI DWDM

38Km38Km

LSU HSC ShreveportLSU HSC ShreveportLONI POPOTM POP

NLR BRNLR BRLONI POP

LSULSU

Jackson, MSJackson, MSLONI DWDM

FranklinFranklinLONI DWDM

SchrieverSchrieverLONI DWDM

Port BarrePort BarreLONI DWDM

13Km13Km

102Km102Km

42Km42Km

LaPlaceLaPlaceLONI DWDM

65Km65Km

CrowleyCrowleyLONI DWDM

Lake CharlesLake CharlesLONI POP

65Km65Km

69Km69Km

11Km11Km

13Km13Km

39Km39Km

LPBLPB11Km11Km

LSULSU

SeminarySeminaryLONI DWDM

GreensburgGreensburgLONI DWDM

TylertownTylertownLONI DWDM

MendenhallMendenhallLONI DWDM

JacksonJacksonLONI DWDM

TallulahTallulahLONI DWDM

StartStartLONI DWDM

EdwardsEdwardsLONI DWDM

RoanokeRoanokeLONI DWDM

35Km35Km38Km38Km

46Km46Km

35Km35Km

38Km38Km

24Km24Km

23Km23Km 64Km64Km 56Km56Km 48Km48Km

LONI Topology – POPs & DWDMs

21SEP06 20Louisiana State Grid

Dick Greenwood

RamahRamahLONI DWDM

LandryLandryLONI DWDM

AlexandriaAlexandriaLONI POP

ULLULLLONI POPOTM POP

CoushattaCoushattaLONI DWDM

DerryDerryLONI DWDM

55Km55Km

80Km80Km

89Km89Km

LSULSU

HammondHammondLONI POP

UNOUNOLONI POP

TulaneTulaneLONI POP

LSU HSC NOLSU HSC NOLONI POPOTM POP

86Km86Km 48Km48Km

54Km54Km

75Km75Km 20Km20Km

70Km70KmSouthernSouthern

DWDM LoopDWDM Loop

MonroeMonroeLONI POP

La TechLa TechLONI POP

82Km82Km

55Km55Km

82Km82Km

72Km72Km

61Km61Km

82Km82Km

Northern DWDM LoopNorthern DWDM Loop

SouthernSouthernLONI POP

MindenMindenLONI DWDM

38Km38Km

LSU HSC ShreveportLSU HSC ShreveportLONI POPOTM POP

NLR BRNLR BRLONI POP

LSULSU

Jackson, MSJackson, MSLONI DWDM

FranklinFranklinLONI DWDM

SchrieverSchrieverLONI DWDM

Port BarrePort BarreLONI DWDM

13Km13Km

102Km102Km

42Km42Km

LaPlaceLaPlaceLONI DWDM

65Km65Km

CrowleyCrowleyLONI DWDM

Lake CharlesLake CharlesLONI POP

65Km65Km

69Km69Km

11Km11Km

13Km13Km

39Km39Km

LPBLPB11Km11Km

LSULSU

SeminarySeminaryLONI DWDM

GreensburgGreensburgLONI DWDM

TylertownTylertownLONI DWDM

MendenhallMendenhallLONI DWDM

JacksonJacksonLONI DWDM

TallulahTallulahLONI DWDM

StartStartLONI DWDM

EdwardsEdwardsLONI DWDM

RoanokeRoanokeLONI DWDM

35Km35Km38Km38Km

46Km46Km

35Km35Km

38Km38Km

24Km24Km

23Km23Km 64Km64Km 56Km56Km 48Km48Km

LONI Topology - Status

21SEP06 21Louisiana State Grid

Dick Greenwood

LONI Computing Power

• At each 6 universities:– IBM p5-575 system(112 power proc, 224 GB

RAM)• At LSU:

– SuperMike (1024 Xeon proc, 1 TB RAM)– SuperHelix (256 Xeon proc, 256 GB

RAM)– MiniMike (32 Xeon proc, 32 GB RAM)– SGI Prism (32 proc, 128 GB RAM)– Nemeaux (64 Mac processors)– LSU Campus Grid with ~2000 processors

(soon)• At UL Lafayette

– LITE (384 proc, 384 GB RAM) • At Latech

– CAPS (30 processors)• And several others..