last update: 21/04/23 12:09

DataGridDataGrid

Overview & StatusOverview & Status

CNAP Meeting 13th October 2000CNAP Meeting 13th October 2000

R.P.Middleton (DataGrid UK Manager)R.P.Middleton (DataGrid UK Manager)

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 2

Rapid OverviewRapid Overview

Proposal favourably reviewed -> now negotiating contract with Proposal favourably reviewed -> now negotiating contract with EUEU

10 Meuro, 21 partners (6 principal) -> BIG consortium10 Meuro, 21 partners (6 principal) -> BIG consortium Collaboration with Earth Observation & Bio-sciencesCollaboration with Earth Observation & Bio-sciences 4 + 1 “funded” posts for UK4 + 1 “funded” posts for UK ( WP3: 3+1; WP5: 1 )( WP3: 3+1; WP5: 1 ) UK Industrial Partner : IBMUK Industrial Partner : IBM PMB - 6 month rotating chair; PTB includes experimentsPMB - 6 month rotating chair; PTB includes experiments CERN Project Office (Project Manager + Architect + Secretariat)CERN Project Office (Project Manager + Architect + Secretariat) Accompanying Measures : “Research & Industry Forum” Accompanying Measures : “Research & Industry Forum” Frequent WP managers phone conferencesFrequent WP managers phone conferences Prototype work already starting (e.g. Globus installation & tests)Prototype work already starting (e.g. Globus installation & tests) Commitment to Open Source for common software componentsCommitment to Open Source for common software components Presented at DANTE/Geant, Terena conference, ECFA, HEP-CCCPresented at DANTE/Geant, Terena conference, ECFA, HEP-CCC High visibility in EU & their IST programmeHigh visibility in EU & their IST programme

(http://europa.eu.int/comm/information_society/eeurope/news/index_en.htm)(http://europa.eu.int/comm/information_society/eeurope/news/index_en.htm)

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 3

So What is it ?So What is it ?

Principal goals:Principal goals: Middleware for fabric & Grid management Linking together of “national” PP Grid infrastructures Large scale testbed - major fraction of one LHC experiment Production quality HEP demonstrations

“mock data”, simulation analysis, current experiments “Other science” demonstrations on testbeds

Three year phased developments & demosThree year phased developments & demos Anticipated start (of EU money) ~Jan 2001

Complementary to other EU GRID projectsComplementary to other EU GRID projects EuroGrid: Uniform access to parallel supercomputing resources

Synergies being developedSynergies being developed Instumental in merge of GridForum & EGRID into Global Grid

Forum Associated Industry and Research Forum (Accompanying

Measure ?) Close contact with Globus team (with informal input to

DataGrid)

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 4

……and What it is Notand What it is Not

No funding for kitNo funding for kit get from national resources (CERN ?)

No substitute for a national Grid programmeNo substitute for a national Grid programme but an external link to LHC data

No money for writing PP specific codeNo money for writing PP specific code EU funds “economic development”, not physics research

Not THE final solution for LHCNot THE final solution for LHC but a very important stepping stone

Not a full-scale implementationNot a full-scale implementation but a realistic prototype

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 5

ParticipantsParticipants

Main partners: CERN, INFN(I), CNRS(F), PPARC(UK), Main partners: CERN, INFN(I), CNRS(F), PPARC(UK), NIKHEF(NL), ESA-Earth Observation NIKHEF(NL), ESA-Earth Observation

Other sciences: KNMI(NL), Biology, Medicine Other sciences: KNMI(NL), Biology, Medicine Industrial participation: CS SI/F, DataMat/I, IBM/UKIndustrial participation: CS SI/F, DataMat/I, IBM/UK Associated partners: Czech Republic, Finland, Associated partners: Czech Republic, Finland,

Germany, Hungary, Spain, Sweden, (- mostly from Germany, Hungary, Spain, Sweden, (- mostly from computer science)computer science)

Formal collaboration with USA being establishedFormal collaboration with USA being established Industry and Research Project Forum with Industry and Research Project Forum with

representatives from:representatives from: Member Countries + Denmark, Greece, Israel, Japan,

Norway, Poland, Portugal, Russia, Switzerland, +...

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 6

WorkpackagesWorkpackages

WP 1 Grid Workload Management (C. Vistoli/Italy)WP 1 Grid Workload Management (C. Vistoli/Italy) WP 2 Grid Data ManagementWP 2 Grid Data Management (B. Segal/CERN) (B. Segal/CERN) WP 3 Grid Monitoring servicesWP 3 Grid Monitoring services (R. Middleton/ (R. Middleton/UKUK)) WP 4 Fabric ManagementWP 4 Fabric Management (T. Smith/CERN) (T. Smith/CERN) WP 5 Mass Storage Management (J. Gordon/WP 5 Mass Storage Management (J. Gordon/UKUK)) WP 6 Integration TestbedWP 6 Integration Testbed (F. Etienne/France) (F. Etienne/France) WP 7 Network ServicesWP 7 Network Services (C. Michau/France) (C. Michau/France) WP 8 HEP ApplicationsWP 8 HEP Applications (F. Carminati/CERN) (F. Carminati/CERN) WP 9 EO Science ApplicationsWP 9 EO Science Applications (L. Fusco/ESA) (L. Fusco/ESA) WP 10 Biology ApplicationsWP 10 Biology Applications (C. Michau/France) (C. Michau/France) WP 11 Dissemination WP 11 Dissemination (G. Mascari/Italy) (G. Mascari/Italy) WP 12 Project ManagementWP 12 Project Management (F. Gagliardi/CERN) (F. Gagliardi/CERN)

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 7

Workpackages - UKWorkpackages - UK

WP 1 Grid Workload Management WP 1 Grid Workload Management A.Martin-QMWA.Martin-QMW WP 2 Grid Data ManagementWP 2 Grid Data Management A.Doyle-GlasgowA.Doyle-Glasgow WP 3 Grid Monitoring servicesWP 3 Grid Monitoring services R.Middleton-RALR.Middleton-RAL WP 4 Fabric ManagementWP 4 Fabric Management A.Sansum-RALA.Sansum-RAL WP 5 Mass Storage Management WP 5 Mass Storage Management J.Gordon-RALJ.Gordon-RAL WP 6 Integration TestbedWP 6 Integration Testbed D.Newbold-BristolD.Newbold-Bristol WP 7 Network ServicesWP 7 Network Services P.Clarke-PPNCG/UCLP.Clarke-PPNCG/UCL WP 8 HEP ApplicationsWP 8 HEP Applications N/A (?)N/A (?) WP 9 EO Science ApplicationsWP 9 EO Science Applications ( c/o R.Middleton-( c/o R.Middleton-

RAL )RAL ) WP 10 Biology ApplicationsWP 10 Biology Applications ( c/o P.Jeffreys-RAL )( c/o P.Jeffreys-RAL ) WP 11 Dissemination WP 11 Dissemination P.Jeffreys-RALP.Jeffreys-RAL WP 12 Project ManagementWP 12 Project Management R.Middleton-RALR.Middleton-RAL

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 8

Simplified Workpackage Simplified Workpackage RelationshipsRelationships

HEP Apps (WP8) EO Apps (WP9) Bio Apps (WP10)

Workload Management (WP1)

Data Management (WP2) Monitoring Services (WP3)

Globus Middleware

Fabric Mangage-ment (WP4)

Networking (WP7)Mass Storage

Management (WP5)

Applications

Data Grid Services

Core Middleware

PhysicalFabric

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 9

UpdateUpdate

Negotiating EU ContractNegotiating EU Contract deadline 25th October 3rd iteration, but slowly converging !! background doc. on Testbed (accepted)

Turn project/management from re-active to pro-activeTurn project/management from re-active to pro-active Forging links : Globus, GriPhyN, PPDG, GF/GGF/EgridForging links : Globus, GriPhyN, PPDG, GF/GGF/Egrid Network negotiation - UKERNA, Dante, etcNetwork negotiation - UKERNA, Dante, etc Search for Project Architect, PTB ChairmanSearch for Project Architect, PTB Chairman Formation of ATFFormation of ATF

Attend GF5, meet with Globus, GriPhyN, PPDG DataGrid “kick-off” in AmsterdamDataGrid “kick-off” in Amsterdam

Joint with GGF1 ? (confirm next week in Boston)

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 10

Organogram ?Organogram ?

PMB

Executive (5 pers)

Arch Task Force

Proj Tech Board

WP1WP1WP1WP1Middleware

WP1-5Testbed & NetworkWP1-5WP1-5HEP/EO/BIO

WP1-5WP11 & 12

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 11

PoliticsPolitics

DataGrid has VERY HIGH profile in EU & Government(s)DataGrid has VERY HIGH profile in EU & Government(s) On lips at OST… Seen (in OST) as THE link to international activity,

standards, etc. Has declared programme bringing in bio- & earth sciences

DataGrid has much support in high placesDataGrid has much support in high places also has critics (enemies) in high places

DataGrid MUST DELIVERDataGrid MUST DELIVER failure is not possible late delivery / poor results ==> acutely embarassing …and LHC computing could fragment if no Grid-scale

solution UK domestic funding taken VERY seriously by DataGrid UK domestic funding taken VERY seriously by DataGrid

partnerspartners even our domestic programme is very exposed !!

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 12

DataGrid : DomesticDataGrid : Domestic

ProblemsProblems JIF held over - was basis for DataGrid planning in UK Ability to attend meetings Testbed (0) and Demos MUST start to deliver ==> ramp up effort

Working with PPARC…no solution yetWorking with PPARC…no solution yet Travel £15k Enabling “unfunded” effort ramp-up to 15 FTE 10% Yr 1 of JIF (CPU & disk) for Jan 2001 £105k

EU posts open for 5 FTE (actually 4.3)EU posts open for 5 FTE (actually 4.3) 1 Mass storage 3 Monitoring Service (consortium requires cluster) 1 HEP Applications (significant time at CERN)

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 13

Jan 2001 @ Tier-1Jan 2001 @ Tier-1

Need Resource NOW which can be dedicated to GridNeed Resource NOW which can be dedicated to Grid no impact on existing services shared internationally

JIF Yr 1 (neglect tape & network)JIF Yr 1 (neglect tape & network) 830 CPU processors (in PC99 units = 450 MHZ PC = 1

SpecInt95) 25 TBytes of disk space

10% Proposal for Jan 200110% Proposal for Jan 2001 20 Dual CPU Pentium processors @ ~£2.5k each £ 50k

(each CPU roughly 2 x PC99 performance) 2 TByte of disk space @ ~£20k per Tbyte £ 40k networking and infrastructure £ 15k

TOTALTOTAL£105k£105k

4-6 weeks of FTE Do not want to continue in 10% size chunks !!Do not want to continue in 10% size chunks !!

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 14

EffortEffort

PPARC PPARCWP Title Unfunded Funded (staff months)1 Workload Management 18 02 Data Management 54 03 Monitoring Services 66 1084 Fabric Management 18 05 Mass Storage Management 54 366 Testbeds & Demonstrations 108 07 Networking 72 08 HEP Applications 144 129 Earth Sciences 0 0 (invisible to EU)10 Bio Sciences 0 0 (3.6 budgeted)11 Dissemination 0 0 (3.6 budgeted)12 Project Management 0 0 (18 budgeted)

TOTAL 534 156

Grand Total 690

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 15

Effort profilesEffort profiles

Middleware (WP1-5)Middleware (WP1-5) 210 staff months210 staff months Testbed/Apps (WP6-8)Testbed/Apps (WP6-8) 324 staff months324 staff months To date…To date…

guess < 4 FTE in use around UK (incl RAL) virtually no Middleware development

…and little sign of effort here from ppUK !! ppUK focussed on deployment

Globus testbed Networking Applications

Ramp-up by Jan 2001 is CRITICALRamp-up by Jan 2001 is CRITICAL

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 16

Marseille Workshop 4-6 Sept 2000Marseille Workshop 4-6 Sept 2000

Rush to setup Sept workshopRush to setup Sept workshop Difficult to organise an agenda over summer Impossible to get people to commit to come during holiday

season However, forward progress ...However, forward progress ...

100 attendees (8 (9) from UK = RAL) Cross-workpackage issues aired Worked on fine detail that timescales had not permitted

before ! “Team Building” for many workpackages

Who are we ? What is there to do and who’s going to do it ?

Mentor feedback - Carl Kesselmann - EXTREMELY usefulMentor feedback - Carl Kesselmann - EXTREMELY useful

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 17

Marseille DecisionsMarseille Decisions

Project ManagementProject Management Internal Reviewer contract (with EPCC)

Project Technical BoardProject Technical Board 2 (per WP) + 4 (HEP) + 4 (EO) + 2 (Bio) + PM 35 members first meeting early October

Architecture Task Force (Project Architect ?)Architecture Task Force (Project Architect ?) Project-wide milestonesProject-wide milestones

focussed, major goals (i.e. take a problem & solve it) internal testbed cycle of 6 months external testbed cycle annually

MeetingsMeetings 4 * DataGrid weeks per year Annual review (incl. deliverables) by external referees Annual project conference (aligned with IST event ?)

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 18

Contract Meeting - 11th Sept 2000Contract Meeting - 11th Sept 2000

No Show-stoppersNo Show-stoppers Very high profile / visibility within EUVery high profile / visibility within EU

seen as a key project within IST programme ! Updates to management structure satisfied EUUpdates to management structure satisfied EU A shopping list of details to fixA shopping list of details to fix

missing deliverables uniformity of deliverables risk analysis dissemination activities with external groups (e.g. standards) Research & Industry forum (Accompanying Measure)

Rumour : EU planning new, specific Grid action in 2001Rumour : EU planning new, specific Grid action in 2001

DataGrid Status - CNAP Meeting 13-Oct-00 - R.P.Middleton13th September 2000 19

TimelineTimeline

JulyJuly Proposal accepted --> Contract NegotiationsProposal accepted --> Contract Negotiations 25th Aug25th Aug CPF & draft Technical Annex to EUCPF & draft Technical Annex to EU 4-6 Sep4-6 Sep Technical Workshop in MarseilleTechnical Workshop in Marseille 11th Sep11th Sep Contract negotiation in BrusselsContract negotiation in Brussels 22nd Sep22nd Sep Revisions submitted to EURevisions submitted to EU this weekthis week More revisions submitted to EUMore revisions submitted to EU . . .. . . 25th Oct25th Oct Last date for contract input !!Last date for contract input !!

(otherwise miss 2000 budget)(otherwise miss 2000 budget) Dec 2000Dec 2000 DataGrid/GriPhyN/PPDG Management DataGrid/GriPhyN/PPDG Management

BoardsBoards(technical collaboration)(technical collaboration)

1st Jan 20011st Jan 2001 Current aim for project startCurrent aim for project start Feb 2001Feb 2001 GGF1 & DataGrid “Kick-off” meetingGGF1 & DataGrid “Kick-off” meeting