ATLAS Grid Activities Preparing for Data Analysis

J. Shank DOSAR Workshop LSU 2 April 2009

DOSAR Workshop VII 2 April 2009

1

ATLAS Grid ActivitiesPreparing for Data Analysis

Jim Shank



Overview

ATLAS Monte Carlo produciton in 2008 Data (cosmic and single beam) in 2008 Production and Distributed Analysis (PandDA)

system Some features of the ATLAS Computing Model

Analysis model for the US Distributed Analysis Worldwide: Ganga/PanDA

and Hammercloud + other readiness tests Tier 3 centers in the US

2



Beam Splash Event

3



First ATLAS Beam Events, 10 Sept. 2008

Data Exports to T1s Throughput in MB/s

Effect of concurrent data access from centralized

transfers and user activity(overload of disk server)

Number of errors

4

CERN Storage system overload. DDM worked. Subsequently we limited user access to the storage system.



December 2008 Reprocessing

5


DOSAR Workshop VII 2 April 2009PanDA production (Monte Carlo

Simulation/Reconstruction) 2008

6

Grouped by Cloud = Tier 1 center + all it’s associated Tier 2 centers



U.S. Production in 2008

7

More than our share—indicates others not delivering their expected levels



8

DDM : Data Replication

Datasets subscription intervalsData replication to Tier-2s

US Tier2s

ATLAS Beam and Cosmics data replication from CERN to Tier-1s and calibration Tier-2s. Sep-Nov 2008

BNL&AGLT2



9

DDM : Data replication between Tier-1s

Functional Test. Tier-1-Tier-1s data replication status. FZK experienced problems with dCache. Data export is affected

Tier-1-Tier-1s and prestaging data replication status. Data reprocessing.All Tier-1s Operational. Red : data transfer completion on 95% (data staging at CNAF)

Torre Wenaus, BNLTorre Wenaus, BNL 1010

PanDA OverviewPanDA Overview

• Launched 8/05 by US ATLAS to achieve scalable data-driven WMS

• Designed for analysis as well as production

• Insulates users from distributed computing complexity

• Low entry threshold

• US ATLAS production since late ‘05

• US analysis since Spring ’06

• ATLAS-wide production since early ‘08

• ATLAS-wide analysis still rolling out

• OSG WMS program since 9/06

• Launched 8/05 by US ATLAS to achieve scalable data-driven WMS

• Designed for analysis as well as production

• Insulates users from distributed computing complexity

• Low entry threshold

• US ATLAS production since late ‘05

• US analysis since Spring ’06

• ATLAS-wide production since early ‘08

• ATLAS-wide analysis still rolling out

• OSG WMS program since 9/06

Workload management system for Production ANd Distributed AnalysisWorkload management system for Production ANd Distributed Analysis



11

Panda/pathena Users

4 million jobs in last 6 months

473 users in last 6 months352 users in last 3 months90 users in last month

271 users with >1000 jobs96 users with >10000 jobs



ATLAS ANALYSIS

12



ATLAS Data Types

Still evolving…

13



ATLAS Analysis Data Flow

14



15



16



17

USUS T2 sites

Analysis Readiness Tests

Nurcan OzturkNurcan Ozturk 18

Ideas for a Stress Test (1)

Initiated by Jim Cochran (US ATLAS Analysis Support Group Chair). Below is a summary of plans from Akira Shibata (March 10th). Goal: Stress testing of the analysis queues in the Tier2 sites with analysis jobs as

realistic as possible both in volume and quality. We would like to make sure that the Tier2 sites are ready to accept real data and analysis queues to analyze them.

Time scale: sometime near the end of May 2009. Outline of this exercise:

To make this exercise more useful and interesting we will generate and simulate (Atlfast-II) a large amount of mixed sample at Tier2’s.

We are currently trying to define the jobs for this exercise and we expect this to be finalized after the BNL jamboree this week.

The mixed sample is a blind mix of all Standard Model processes, which we call "data" in this exercise.

For the one day stress test, we will invite people with existing analysis to try and analyze the data using Tier2 resources only.

We will compile a list of people who have the ability to participate.

Nurcan OzturkNurcan Ozturk 19

Ideas for a Stress Test (2)

Estimate of data volume: A very rough estimate of the data volume is 100M-1B events. Assuming 100kB/event (realistic considering no truth info and no trigger info), this sets an upper limit of 100TB in total (split among 5 Tier2’s). This is probably an upper-limit from the current availability of USER/GROUP disk on Tier2 (which is in addition to MC/DATA/PROD and CALIB disk).

Estimate of computing capability: There are "plenty" of machines assigned to analysis though the current load of analysis queue is rather low. The computing nodes are usually shared between production and analysis and typically configured with upper limit and priority. For example MWT2 has 1200 cores and setup to run analysis jobs with priority with an upper limit of 400 cores. If production jobs are not coming in, the number of running analysis jobs can exceed this limit.

Site configuration: Site configuration varies among the Tier2 sites. We will compile a table showing configuration of each analysis queue; direct reading versus local copying, xrootd versus dcache, etc. We will compare the performance of queues based on their configuration.



Four Types of Tier 3 Systems

T3gs T3 with Grid Services Details in next slides

T3g T3 with Grid Connectivity details in next slides

T3w Tier 3 Workstation

unclustered workstations...OSG, DQ2 client, root, etc

T3af Tier 3 system built into lab or university analysis facility

20



21



22



Conclusions

Monte Carlo Simulation/Reconstruction working

well world wide with PanDA submission system Data reprocessing with PanDA working, but need

further tests of file staging from tape. Analysis Model still evolving

In the U.S., big emphasis on getting T3’s up and

running Analysis stress test coming in May-June

Ready for collision data in late 2009

23



Backup

24


PanDA OperationPanDA Operation

T. MaenoT. Maeno

DatamanagementDatamanagement

ATLAS productionATLAS production

AnalysisAnalysis


PanDA Production Dataflow/WorkflowPanDA Production Dataflow/Workflow


Analysis with PanDA: pathenaAnalysis with PanDA: pathena

Tadashi MaenoTadashi Maeno

Running the ATLAS software:Locally: athena <job opts>PanDA: pathena --inDS --outDS <job opts>

Running the ATLAS software:Locally: athena <job opts>PanDA: pathena --inDS --outDS <job opts>

Outputs can be sent to xrootd/PROOF farm, directly accessible for PROOF analysis

Outputs can be sent to xrootd/PROOF farm, directly accessible for PROOF analysis

Documents

ATLAS Grid Activities Preparing for Data Analysis