April 18, 20231

GrenchMark:A synthetic workload generator for Grids

KOALA Workshop

A. Iosup, H. Mohamed, D.H.J. EpemaPDS Group, ST/EWI, TU Delft

April 18, 20232

About productivity and efficiency…

We now have

a good scheduler for the DAS (Koala)

How do we make use of it every day?

April 18, 20233

Productivity, efficiency

1. Run 1 jobrun 1 command ET (every time)

April 18, 20234

Productivity, efficiency

1. Run 1 jobrun 1 command ET (every time)

2. Run 10 jobs write 10 lines script @1 (once), run 1 command @1

April 18, 20235

Productivity, efficiency

1. Run 1 jobrun 1 command ET (every time)

2. Run 10 jobs write 10 lines script @1 (once), run 1 command @1

3. Run 100 jobs, parameterize runs write 25 lines script @1, run 1 command @1

April 18, 20236

Productivity, efficiency

1. Run 1 jobrun 1 command ET (every time)

2. Run 10 jobs write 10 lines script @1 (once), run 1 command @1

3. Run 100 jobs, parameterize runs write 25 lines script @1, run 1 command @1

4. (Re-)Run (the same) 100 jobs, parameterize runs write 500 lines script @1, run 1 command @1

April 18, 20237

Productivity, efficiency

6. Run “100 times each of: 100 jobs, parameterized, 1 to 5 components per job, preference for 2 and 4 components, 2 to 128 processors per job, preference for 2, 4, 16 and 32 processors, arrival rate Poisson or Alpha-Beta or mixed, stop runs after 15 or 60 minutes, …” write 5000 lines script @1, run 1 command @1

April 18, 20238

Productivity, efficiency

6. Run “100 times each of: 100 jobs, parameterized, 1 to 5 components per job, preference for 2 and 4 components, 2 to 128 processors per job, preference for 2, 4, 16 and 32 processors, arrival rate Poisson or Alpha-Beta or mixed, stop runs after 15 or 60 minutes, …” write 5000 lines script @1, run 1 command @1

7. … and this was per test / project / article

April 18, 20239

Problem statement (revisited)

• Synthetic worklods are the basis of grid schedulers comparison [our work], and of ...• Functionality testing in the DAS• (Early) performance testing for

application development

• Comparing alternative settings for the DAS• Many others …

• Need a standard tool for synthetic workloads generation and submission

April 18, 202310

The GrenchMark framework

• What’s in a name?grid benchmark → help standardizing the testing procedure, but benchmarks are too early…

• GrenchMark• Easy-to-use tools to create synthetic workloads• Easy-to-use tools to submit synthetic workloads• Flexible, portable, extensible

April 18, 202311

The GrenchMark framework

April 18, 202312

The GrenchMark process: Describe

$ cat wl-desc.in# File-type: text/wl-spec#Times UnitType SiteType SiteInfo ArrivalTimeDistr OtherInfo 50 ssert non-fixed *:* U(0,100) - 10 cpi-gm fixed fs1;…Poisson(60) NProc=1;2;4;… 35 cpi-g2 fixed *:* Poisson(60) ExternalFile=

my.data

• wl-desc.in describes the workload to be generated

$ cat my.data NComp=10/50; 4/20; 2/10; 5/10; 3 NProc=1/10; 2/10; 4/10; 8/10; 16/10; 32/10; 3;… …

• my.data describes the characteristics of a part of the workload

April 18, 202313

$ ./wl-gen.py wl-desc.in

The GrenchMark process: Generate

• Build directory structure• Generate Times x Jobs JDFs

2• Sample run:

• 95 jobs / 411 components• 95 files / 32 directories• 300KB data

April 18, 202314

The GrenchMark process: Submit

$ ./wl-gen.py wl-desc.in $ ./wl-submit.py out/wl-to-submit.wl 3

3

TODO

• Sample run:• multi-threaded – submission delay +/- 0.01s• 95 JDFs

April 18, 202315

GrenchMark status

• Already done in Python• Generator + Globus, KOALA generators + RSL printer• Submitter (Koala, Globus, …)• Results analyzer (crude)

• Support for applications• Unitary, 3 types: sequential, MPI, Ibis (Java)• +35 different applications• Composite applications: DAG-based

• Ongoing work• Automated results analyzer• Better modeling capabilities

April 18, 202316

Extending GrenchMark

Motto:Extending GrenchMark is easy!

1. More workload generators• a function with a predefined name in a Python

module• auto-loaded

2. Interface with C/C++, Ruby, Perl, Java, …• define your own protocol and write the Python end

3. Interface with other systems (SGE, …)• a function with a predefined name in a Python

module• auto-loaded

April 18, 202317

GrenchMark for testing KOALA…

• KOALA

• Non-fixed, fixed, semi-fixed jobs

• Test & evaluate• Status: pre-release, supposed stable• Workloads with different jobs requirements,

inter-arrival rates, co-allocated v. single site jobs…• Jobs success rate, KOALA’s overhead and bottlenecks

• Results• 2 major bugs first day, +10 bugs overall (all fixed)

• KOALA is officially released (full credit to KOALA developers, 10x for testing with GrenchMark)

+5,000 jobs successfully

run

April 18, 202318

… and DAS-2’s functionality …

• Already done• Evaluate for KOALA + Globus + DAS-2

• jobs success rate, turnaround time, middleware overhead, types and sources of errors

• Sample results• 5 workloads• 500 jobs

A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema,

Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, In Proc. of CoreGRID IW, Nov 2005.

+1,000 jobs successfully

run

April 18, 202319

… and for analyzing, testing, and comparing grids

• Grid system analysis• Performance testing / analysis• What-if analysis

• Functionality testing in grid environments• System functionality testing• Periodic testing

• Comparing grid settings• Single-site vs. co-allocated jobs• Unitary vs. composite jobs

A.Iosup, D.H.J.Epema, GrenchMark: A Framework for Analyzing, Testing, and Comparing Grids, In CCGrid 2006 (accepted).

+2,500 jobs successfully

run

April 18, 202320

Conclusions

• GrenchMark generates and submits synthetic workloadseasy-to-use, flexible, portable, extensible, …

• Experienceused GrenchMark to test KOALA’s functionality and performance. used GrenchMark to test some DAS/Grid functionality. +5,000 jobs generated and run … and counting.

• … advertisementHave specific Grid applications types you would like to test? Test with GrenchMark!

April 18, 202321

Thank you!

Questions? Remarks? Observations? All welcome!

Grenchmark http://grenchmark.st.ewi.tudelft.nl/ [ “grenchmark”]

Alexandru IOSUPTU Delft

A.Iosup@tudelft.nl[ “iosup”]

Also: H. Mohamed and W. Lammers (KOALA), K. Verstoep and P. Anita (DAS admins), R. van Nieuwpoort and J. Maassen (Ibis)