McClatchey draft2

7/31/2019 McClatchey draft2

1/33

Deployment of a GridDeployment of a Grid--

based Medical Imagingbased Medical ImagingApplicationApplication

HealthGridHealthGrid05 Conference, Oxford UK April 200505 Conference, Oxford UK April 2005

RichardRichard McClatcheyMcClatcheyMammoGrid Technical Coordinator, UWEMammoGrid Technical Coordinator, UWE


2/33

7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 2

ContentsContents

MammoGrid challengesMammoGrid challenges

Approach andApproach and technolgiestechnolgies adoptedadoptedMammoGrid prototypingMammoGrid prototyping

Services and Grid solutionsServices and Grid solutions

Deployment and future workDeployment and future work

Lessons learned and conclusionsLessons learned and conclusions


3/33


AboutAbout MammogridMammogrid

EU FP5 funded project, start: 2002, end:2005EU FP5 funded project, start: 2002, end:2005

To build a gridTo build a grid--powered panpowered pan--europeaneuropeanmammography database and evaluate its usemammography database and evaluate its usein a clinical environmentin a clinical environment

Participants:Participants: Medical Institutions:Medical Institutions:

UdineUdine University Hospital;University Hospital; AddenbrookesAddenbrookes

Hospital, Cambridge; Oxford UniversityHospital, Cambridge; Oxford University Academic Institutions, commercial partners:Academic Institutions, commercial partners:

UWE, CERN, Mirada Solutions Ltd, Pisa,UWE, CERN, Mirada Solutions Ltd, Pisa, SassariSassari


4/33


5/33


Technical ChallengesTechnical Challenges

Make GridMake Grid work in practicework in practice andand usableusable in hospitalsin hospitals(execute clinical queries, run SMF &(execute clinical queries, run SMF & CADeCADe, share, share

diagnoses)diagnoses)Provide aProvide a distributed and federateddistributed and federated clinical dataclinical datamanagement systemmanagement systemDeliver aDeliver a secure systemsecure system that could be integrated into athat could be integrated into ahospital information systemhospital information systemInvestigate how the medical application can be isolatedInvestigate how the medical application can be isolatedfrom the (still evolving) Grid as new Grid flavoursfrom the (still evolving) Grid as new Grid flavours

emerge (e.g. OGSIemerge (e.g. OGSI-->Web Services/Grid Services).>Web Services/Grid Services).ProvideProvide persistencypersistency for data models and metafor data models and meta--datadatamodelsmodels to handleto handle heterogeneityheterogeneity


6/33


Grid challenges:Grid challenges:

interoperabilityinteroperabilityLegal restrictions on access to dataLegal restrictions on access to data Clinicians, researchers, developers, Govt,Clinicians, researchers, developers, Govt,

Data resides in hospitalsData resides in hospitals Existing systems, firewall protectedExisting systems, firewall protected

Combining several databasesCombining several databases

HeterogeneityHeterogeneity

Secure file transfer essentialSecure file transfer essential

Large images to be transferredLarge images to be transferred

Approx 100Mb per patientApprox 100Mb per patient

APIs needed for third party applicationsAPIs needed for third party applications


7/33


ContentsContents







8/33


ImplementationImplementation

ApproachApproachApplyApply emerging GRID technologyemerging GRID technology rather than develop it.rather than develop it.AA lightweightlightweight GRID, study its usage in hospitalsGRID, study its usage in hospitals

AA prototype federated databaseprototype federated database of mammograms inof mammograms inhospitals in the UK and Italyhospitals in the UK and ItalyIt will investigate :It will investigate :

metameta--data for resolvingdata for resolving queriesqueries standardisedstandardised mammograms to resolve image variabilitymammograms to resolve image variability Health dataHealth data securitysecurity using a novelusing a novel Grid boxGrid box

thethe infrastructureinfrastructure needed forneeded for CADeCADeIt will provideIt will provide feedbackfeedback from the Hospital communityfrom the Hospital communityTo inform the next generation of HealthGridsTo inform the next generation of HealthGrids

P


9/33


Steps in ProjectSteps in Project

P1Centralised DB

(Oct 2003)

P1Distributed DB

(April 2004)

P1.5Web services

(Nov 2004)

P1.5GRID-enabled

services

P2Multiple VOs

(Post MammoGrid)

EGEE

MammoGrid

AliEn

Stack of

glite

EDG

AliEn

we are

hereTime

U d i iU d i i


10/33


UnderpinningUnderpinningtechnologiestechnologies

Distributed computationsDistributed computations Web Services (SOAP, XML,WSDL);Web Services (SOAP, XML,WSDL);

ServiceService--Oriented ArchitectureOriented Architecture

(SOA);(SOA);

AliEn Grid MiddlewareAliEn Grid Middleware

EGEEEGEE gLitegLite, the emerging Grids, the emerging Gridsstandardstandard

MammographyMammography DICOM:DICOM:

DICOM files are transferred overDICOM files are transferred overSCPSCP--SCU;SCU;

DICOM files are stored asDICOM files are stored asimmutable objects.immutable objects.

SMF:SMF: Standardized Mammography FormStandardized Mammography Form

(Mirada)(Mirada)

CADe:CADe: ComputerComputer--Aided DetectionAided Detection

algorithms (CALMA)algorithms (CALMA)


11/33


ContentsContents







12/33


MammogridMammogrid PrototypesPrototypes

Prototype 1.0Prototype 1.0 Prototype 1.5Prototype 1.5 Prototype 2.0Prototype 2.0No metadataNo metadata

Centralized dataCentralized data

Patient data storedPatient data storedalongside filesalongside files

Services areServices areinstantiated insideinstantiated inside AliEnAliEn

Decentralized metadataDecentralized metadataand dataand data

File storage separatedFile storage separatedfrom semifrom semi--structured datastructured data

Query semanticsQuery semantics

Query PerformanceQuery Performance

Portal andPortal and QueryManagerQueryManager::staticstatic webserviceswebservices

gLitegLite interface: static,interface: static,

insecureinsecure webserviceswebservices

GAS factoryGAS factoryto secureto secure gLitegLite UIUI

Portal andPortal andQueryManagerQueryManager: proper: propergrid services (OGSI)grid services (OGSI)

[Authentication to[Authentication to gLitegLitevia delegation]via delegation]

Query optimizationQuery optimization

As time/resourcesAs time/resourcespermitpermit


13/33


MII: P1.5 OverviewMII: P1.5 Overview

Scanner

MAS

MWS

MG Services

Add Dicom FileQuery DataRetrieve Dicom FileJob Execution

Clinician

CambridgeNode

Oxford Node

Scanner

MAS

MWS

MG Services

Add Dicom FileQuery DataRetrieve Dicom FileJob Execution

Clinician

UdineNode

Local gLite Services

Storage ElementComputing ElementFile TransferMonitoring

Local gLite Services

Storage ElementComputing ElementFile TransferMonitoring

Cambridge Repository

MG data

MG meta-data

Udine Repository

MG data

MG meta-data

VirtualRepository

Central gLite Services

AuthenticationDatabase ProxyTransfer SchedulerJob Scheduler

gLite Repository

Grid dataFile Catalogue

Wh t h bWh t h s b n


14/33


What has beenWhat has been

delivered (P1.5)delivered (P1.5)

A set ofA set of tested and configuredtested and configured GridboxesGridboxes deployed at thedeployed at thehospitals.hospitals.

NearNear--final version of thefinal version of the Mirada Acquisition System andMirada Acquisition System andMammogram WorkstationMammogram WorkstationSuccessful implementation and testingSuccessful implementation and testing of MammoGrid Servicesof MammoGrid Servicesacross deployed sitesacross deployed sites

SMF &SMF & CADeCADe algorithms deployedalgorithms deployed, tested (SMF &, tested (SMF & CADeCADecompatibility)compatibility)P1.5P1.5 gLite/AliEngLite/AliEn--basedbased, distributed database system, distributed database systemWebWeb--service implementationservice implementation with persistently stored data andwith persistently stored data and

metameta--data modelsdata modelsGrid for job and file distribution,Grid for job and file distribution, database outside the Griddatabase outside the GridmiddlewaremiddlewareTest VO at CERN: gLiteTest VO at CERN: gLite--based, MG services are Grid servicesbased, MG services are Grid services

G dbG idb


15/33


GridboxesGridboxes

Design, construction andDesign, construction anddelivery completed by subdelivery completed by sub--contractorcontractor

All units tested andAll units tested anddeployeddeployed

In use atIn use at UdineUdine, Oxford, Oxford

UnivUniv, CERN and Cambridge, CERN and CambridgeConnected to the Grid andConnected to the Grid andavailable for clinicalavailable for clinical

evaluation.evaluation.

G d N dM G id N d


16/33


MammoGrid NodeMammoGrid Node

MammoGridAcquisition Station

deployed in Hospitals & runningInterface of clinicians w/ system

Gridbox

Grid

built & testedhosts Grid engine andMG medical services

d l l


17/33


Cambridge Clinical SiteCambridge Clinical Site


18/33


ContentsContents






M idM id S iS i


19/33


MammogridMammogrid ServicesServices

GridBoxGridBox to Client:to Client: PortalPortal DICOM SCPDICOM SCP

HTTP(G) (retrieve)HTTP(G) (retrieve)

GridBoxGridBox to peers:to peers: Query ManagerQuery Manager

SESE CECE FTDFTD

CentralCentral toto GridBoxGridBox:: LDAP (configure)LDAP (configure) Proxy (Database)Proxy (Database) TransferManagerTransferManager JobManagerJobManager

OptimizersOptimizers

GRIDBOXCLIENT

DICOM SCU/SCP

SOAP over HTTPG

HTTPG/HTTP

DICOM SCP

PORTAL gLite/AliEnServices

QUERYMANAGER

Data

GRIDBOX

DICOM SCPPORTAL

gLite/AliEnServices

QUERYMANAGER

Data

CERN

FILECATALOGUE

CENTRALSERVICES

LDAP

D t D i S hD t D i S h


20/33


Data Design SchemaData Design Schema

SE SE

gLite CENTRAL

MAMMO METAMAMMO DATA


PORTAL PORTAL

PORTAL PORTAL


SE SE


FILE CATALOGUE

Single Entry point: PortalSingle Entry point: Portal

Distributed patient andDistributed patient and

pathology datapathology data

Distributed metadataDistributed metadata(data description)(data description)

Data and schema autonomyData and schema autonomy(wrappers, mediators)(wrappers, mediators)

Centralized File CatalogueCentralized File Catalogue

Distributed File RepositoryDistributed File Repository(Storage Elements)(Storage Elements)

Example: add newExample: add new


21/33


Example: add newExample: add newMammogramMammogram

MAMMOWORKSTATION

1.

Scanner

DICOM SCP2.

3. add()

MAMMOGRIDDATA

4.

MAMMOGRIDMETA-DATA

5.

6.

PORTAL

File

Catalogue,SE

1.1. Clinician Scans MammogramClinician Scans Mammogram

2.2. Client Sends DICOM file toClient Sends DICOM file toSCP buffer inSCP buffer in GridboxGridbox

3.3. Client contacts portal,Client contacts portal,requests add()requests add()

4.4. Portal contactsPortal contacts gLitegLite, inserts, insertsDICOM file into catalogueDICOM file into catalogue

5.5. Portal readsPortal reads MammogridMammogrid metameta--data structuredata structure

6.6. Portal sends data toPortal sends data to

DBBackendDBBackend (XML), data goes(XML), data goestoto MySQLMySQL DBDB

E lE l


22/33


Example: queryExample: query

MAMMOWORKSTATION

1. query()

MAMMOGRIDDATA

MAMMOGRIDMETA-DATA

5.

PORTAL

QUERY

MANAGER

QUERY

HANDLER

2. 6

.

4.

QUERY

MANAGER

QUERY

MANAGER

3.

3.

AGE > 50

FINDING = MASS

MALIGNACY=HIGH

PROBABILITY OFCANCER > 80%

1.1. Clinician constructs queryClinician constructs queryusing GUIusing GUI

2.2. Client sends query XML toClient sends query XML toPortalPortal

3.3. Portal sends query to aPortal sends query to aQuery ManagerQuery Manager

4.4. Query Manager plans queryQuery Manager plans querydistributiondistribution

5.5. Query Manager distributesQuery Manager distributesthe Querythe Query

6.6. Query Handlers receive theQuery Handlers receive thequeryquery

7.7. Query Handler resolvesQuery Handler resolvesagainst metadata DBagainst metadata DB

8.8. Query Handler executesQuery Handler executesquery on thequery on the MammogridMammogrid DBDB

(wrappers, mediators omitted)(wrappers, mediators omitted)

gLite/AliEngLite/AliEn servicesservices


23/33


gLite/AliEngLite/AliEn servicesservicesoverviewoverview

PORTAL

SOAP

HTTP

UI

SE

MG Interface

FTD

CE/ClusterMonitor

aliprod

mgprod

mgprod aliprod

aliprod

mgprod

udineudine

gLite/AliEn site services

MG CA certificate

SSH key

SSH key

SSH key

gLite/AliEnCENTRALSERVICES

Roles:

adminaliprodudinecambridgeoxford

People:

mgadminmgprod (each site )udinecambridgeoxford

adminmgadmin

(HTTP)

+ls()+whereis()

+rm()+submit()

+mkdir()+mirror()

+get()

AliEn::Service::Interface

AliEn::Service

AliEn::UI

Al orithms JobAlgorithms Job ExecExecnn


24/33


Algorithms, JobAlgorithms, Job ExecExecnn..

Algorithms are deployed centrallyAlgorithms are deployed centrallyPresently two different kinds of algorithms:Presently two different kinds of algorithms: SMF:SMF:

Mammogram standardizationMammogram standardization DICOM image into DICOM imageDICOM image into DICOM image

CADeCADe Feature detection (calcification, mass)Feature detection (calcification, mass) DICOM image into text output (later into DICOM SR)DICOM image into text output (later into DICOM SR)

Workflow:Workflow: User selects from available algorithmsUser selects from available algorithms User selects input files (usually result of a query)User selects input files (usually result of a query)

User specifies additional algorithm parametersUser specifies additional algorithm parameters

Portal createsPortal createsjdljdl, submits job (batch, splitting, submits job (batch, splitting)) Monitoring of job statusMonitoring of job status User may examineUser may examine stdoutstdout,, stderrstderr, view and register resulting, view and register resulting

DICOMDICOMJobs running on grid do not access the databaseJobs running on grid do not access the database

MammogridMammogrid andand gLitegLite


25/33


MammogridMammogrid andand gLitegLite

Data Management:Data Management: SemiSemi--structured data management is limited, nonstructured data management is limited, non--

performantperformant..

For our purposes data management is provided by a separateFor our purposes data management is provided by a separatelayerlayer Links between fileLinks between file--based and semibased and semi--structured data:structured data: LFNsLFNs

File Transfers:File Transfers: Unacceptable performance for generic useUnacceptable performance for generic use--cases (1 min forcases (1 min foractual transfer with ~5 min overhead)actual transfer with ~5 min overhead)

A method installed to bypass FTD transfersA method installed to bypass FTD transfers

Job Execution:Job Execution: Problems with splitting (mostly resolved)Problems with splitting (mostly resolved) Job status reporting not reliableJob status reporting not reliable

C


26/33


ContentsContents




Deployment and future workDeployment and future workLessons learned and conclusionsLessons learned and conclusions

MammoGrid solutionMammoGrid solution


27/33


MammoGrid solutionMammoGrid solution

ServiceService--Oriented InfrastructureOriented Infrastructurethat federates autonomouthat federates autonomoumammogram databases on top of a Grid middlewaremammogram databases on top of a Grid middleware..

Medical imaging (MI) service layer:Medical imaging (MI) service layer: Generic services for handling imageGeneric services for handling image--relatedrelated

data (parsing, transforming, storing etc);data (parsing, transforming, storing etc);

Specific imaging services (image analysis,Specific imaging services (image analysis,query services etc.)query services etc.)

GridGrid--aware service layer:aware service layer: Mediates between MI services andMediates between MI services and

underlying Grid middleware;underlying Grid middleware; Examples: storing/retrieving files on theExamples: storing/retrieving files on the

Grid, authentication, job submission andGrid, authentication, job submission andexecution Grid MiddlewareGrid Middleware

e.g.e.g. gLitegLite

GridGrid--aware Servicesaware Services

MedicalMedical--Image ServicesImage Services

ClientsClients

SOA

execution

Prototype MammographyPrototype Mammography


28/33


Prototype MammographyPrototype MammographyDatabaseDatabase

IDC

MAMMOGRIDVO

AddenbrooksHospital,

Cambridge

OxfordUniversity

UdineUniversity

Hospital

M mm ridMammogrid VO st tusVO status


29/33


MammogridMammogrid VO statusVO status

Production P1.5 deployed end of 2004 with emptyProduction P1.5 deployed end of 2004 with emptydata storedata store

P1.5 demonstrated at the 2004 EU Annual TechnicalP1.5 demonstrated at the 2004 EU Annual TechnicalReviewReview

Presently ~2000 images uploaded from previouslyPresently ~2000 images uploaded from previously

acquired data (~20GB)acquired data (~20GB)Batch upload (and batch SMF) is ongoing, liveBatch upload (and batch SMF) is ongoing, liveacquisition startedacquisition started

Clinical trials are starting.Clinical trials are starting.

Final activities:Final activities: bugfixingbugfixing, clinical trials,, clinical trials,

dissemination, commercializationdissemination, commercialization

C t tC t t


30/33


ContentsContents




Deployment and future workDeployment and future workLessons learned and conclusionsLessons learned and conclusions

MammoGrid Future PlansMammoGrid Future Plans


31/33


MammoGrid Future PlansMammoGrid Future Plans

P1.5 Architecture delivered & deployedP1.5 Architecture delivered & deployed P1.5 already present in Oxford and CERNP1.5 already present in Oxford and CERN

Final deployment in Cambridge &Final deployment in Cambridge & UdineUdine hospitalshospitals

20042004

--2005 testing phase2005 testing phase

P2 Architecture under development (resources limited)P2 Architecture under development (resources limited) New Services to provideNew Services to provide

enhanced security and confidentialityenhanced security and confidentiality

legal and ethical conformancelegal and ethical conformance

Facility to addFacility to add

further image and nonfurther image and non--image handling servicesimage handling services

MammoGrid dedicated MetaMammoGrid dedicated Meta--Data DatabaseData Database heterogeneous schema discovery / queryheterogeneous schema discovery / query servicesservices

Framework 6Framework 6 project(sproject(s) to research biomedicine) to research biomedicine

See http://See http://mammogrid.vitamib.commammogrid.vitamib.com

Lessons LearnedLessons Learned -- GridsGrids


32/33


Lessons LearnedLessons Learned -- GridsGrids

GRID isGRID is newnew :: new skills required and few with them.new skills required and few with them.

GRID isGRID is slowslow :: therefore initially concentrate ontherefore initially concentrate onfunctionality rather than performance.functionality rather than performance.

GRID isGRID is changingchanging :: have to hit a moving target.have to hit a moving target.

GRID isGRID is complexcomplex :: many layers of software, developedmany layers of software, developedby different groups.by different groups.

GRID is stillGRID is still insecureinsecure / unreliable / over/ unreliable / over--hypedhypedGRID isGRID is multimulti--flavouredflavoured,, needs standardsneeds standards

Lessons for HealthGridsLessons for HealthGrids


33/33


Lessons for HealthGridsLessons for HealthGrids

Clinician community is highlyClinician community is highly riskrisk--averseaverseMedical dataMedical data anan ethical and legal minefieldethical and legal minefield

Grid software is stillGrid software is still highly volatilehighly volatile

Companies not yet ready to invest inCompanies not yet ready to invest in HealthgridsHealthgridsNo real crossNo real cross--over withover with bio/geneticbio/genetic informationinformation

SoSoResearch neededResearch needed in biomedical applications within biomedical applications withcompanies (recent FP6 call?)companies (recent FP6 call?)

Progress onProgress on Grid standardisationGrid standardisation essential toessential toenableenable outout--ofof--thethe--boxbox capability and stabilitycapability and stability

Ethical issuesEthical issues must be resolved before we getmust be resolved before we get

clinician and companyclinician and company buybuy--inin..

Documents

McClatchey draft2