University of California, San Diego Structural Engineering

SAN DIEGO SUPERCOMPUTER CENTER University of Zürich, Switzerland

Image Guided Therapy ProgramBrigham and Women’s Hospital, Harvard Medical School

PROBLEM: Neuro-surgeons seek to remove as much tumor tissue as possible while minimizing removal of healthy brain tissue

Brain deforms during surgery

Surgeons must align preoperative brain image with intra-operative images to provide surgeons the best opportunity for intra-surgical navigation

A Novel Grid Architecture Integrating Real-Time Data and Intervention During Image Guided Therapy

Radiologists and neurosurgeons at

Brigham and Women’s Hospital, Harvard Medical School transfer 30/40 MB brain images (generated during surgery) to SDSC and other HPC centers for simulation using the MCP scheduler

Parallel Finite element simulation of biomechanical model for volumetric deformation performed on HPC machine; output results are sent to BWH where updated images are shown to surgeons during surgery

Transmission repeated every hour during 6-8 hour surgery

Transmission and FEM simulation must take on the order of minutes

Scheduling and On-Scheduling and On-demand Computingdemand Computing ::

• MCP scheduler MCP scheduler submits the same job to submits the same job to multiple machinesmultiple machines

• As soon as one job As soon as one job starts MCP kills starts MCP kills redundant jobs redundant jobs

Data transfer :Data transfer :Globus-url-copy and SRBGlobus-url-copy and SRB

NSF-ITR Grant NSF-ITR Grant 0427183 – SDSC, UCSDK. Baldridge, A. Majumdar, D. Choi, A. K. Baldridge, A. Majumdar, D. Choi, A. Birnbaum (SDSC), Petr Krysl (UCSD), A. Birnbaum (SDSC), Petr Krysl (UCSD), A. Trivedi (gradstudent)Trivedi (gradstudent)

NSF-ITR Grant 0426558 – Brigham and NSF-ITR Grant 0426558 – Brigham and Women’s Hospital, HarvardWomen’s Hospital, HarvardS. K. Warfield, R. Kikinis, N. Archip (postdoc), S. K. Warfield, R. Kikinis, N. Archip (postdoc), S. Haker, neurosurgeons, radiologistsS. Haker, neurosurgeons, radiologists

Data transfer :Data transfer :Globus-url-copy and SRBGlobus-url-copy and SRB

Timing Results of Current Parallel Finite Element Code

-10.00

20.0030.00

40.0050.00

60.00

1 2 4 8 16 32

# of CPUs

Tim

e (s

ec)

IBM Power3

Itanium2 TeraGrid

IBM Power4

Brain shiftBrain shift

University of California, San Diego Structural Engineering

SAN DIEGO SUPERCOMPUTER CENTER University of Zürich, Switzerland

Image Guided Therapy ProgramBrigham and Women’s Hospital, Harvard Medical School

Famuls FEM AMR deformation Famuls FEM AMR deformation

simulation using elastic solver; simulation using elastic solver; additional tetra-hedra, reducing additional tetra-hedra, reducing error, in the interiorerror, in the interior

Fusion of pre-op fMRI (green, red, Fusion of pre-op fMRI (green, red, yellow), pre-op MRI, and intra-op yellow), pre-op MRI, and intra-op MRI; tumor is dark round region atMRI; tumor is dark round region atthe lower right bottom the lower right bottom

On-demand resource:On-demand resource:Experiment on SDSC Experiment on SDSC and NCSA TG clusterand NCSA TG clusterover 3 day period forover 3 day period forrequest of 5 mins timerequest of 5 mins timefrom 2 to 64 CPUs. Jobfrom 2 to 64 CPUs. Jobwas terminated if it was terminated if it didn’t start in10 minsdidn’t start in10 minsand next request and next request processed. Total of 600processed. Total of 600jobs submitted; ~50 forjobs submitted; ~50 foreach bar on the plots.each bar on the plots.

Results I:Results I: both clusters both clustersshow decreasing likeli-show decreasing likeli-hood of success with hood of success with increasing # of requests.increasing # of requests.

Results II: Results II: relationshiprelationshipbetween the size of requestbetween the size of requestand the length of queue and the length of queue delaydelay

Additions to the ITR projectAdditions to the ITR project

•Initial small funding from IBM (I3) – innovation awardInitial small funding from IBM (I3) – innovation award

•Grad student summer intern ( Prf. D. Gannon, Indiana U.)Grad student summer intern ( Prf. D. Gannon, Indiana U.)

•Undergrad from Berkeley – REU summer scholarshipUndergrad from Berkeley – REU summer scholarship

• Multiple talks, SC demos, presentation, interest from Multiple talks, SC demos, presentation, interest from other groupsother groups

Parallel FEM roadmapParallel FEM roadmap

• Viscoelastic, viscoplastic: constitutive equationsViscoelastic, viscoplastic: constitutive equations

• Implement parallel sparse solver; domain Implement parallel sparse solver; domain decomposition of Famuls FEM codedecomposition of Famuls FEM code

•Accuracy at the MRI voxel level – orders of Accuracy at the MRI voxel level – orders of magnitude increase in data, compute, networkmagnitude increase in data, compute, network

• Dynamic multiscale image updateDynamic multiscale image update

Grid Infrastructure roadmapGrid Infrastructure roadmap

• Software to experiment with queue Software to experiment with queue wait time, data transfer, network -wait time, data transfer, network -flooding versus selective approachflooding versus selective approach

• security (patient privacy, encoding), security (patient privacy, encoding), data (meta-data, archive), visualizationdata (meta-data, archive), visualization

• NMI workflow integrationNMI workflow integration

ITR ITR

DDDASDDDAS

Major cyberinfrastructure Major cyberinfrastructure research: research:

100s of ORs and 10s of TFLOP 100s of ORs and 10s of TFLOP centerscenters

Progress Todate and Future Research Progress Todate and Future Research DirectionDirection