SA2 – Quality Assurance

EMI is partially funded by the European Commission under Grant Agreement RI-261611

SA2 – Quality AssuranceAlberto AIMAR (CERN)SA2 Leader

EMI Second EC Review12 June 2012, Brussels

EMI I

NFSO

-RI-2

6161

1

2

ContextObjectives and Recommendations

Impact on EMI Quality • Build and Test

(ETICS)• Integration Infrastructure

(EMI Testbed)• Metrics• Quality Control

Conclusions

Outline

12/06/2012 EMI 2nd EC Review

https://bit.ly/emisa2

EMI I

NFSO

-RI-2

6161

1

EMI 2nd EC Review 3

Quality Assurance in EMI (SA2)

12/06/2012

EMI I

NFSO

-RI-2

6161

1

4

DoW Objectives


DoW SO 1.4• Common software quality assurance policies and metrics

for all engineering activities• Monitoring metrics and trends quality control activities

value trends, reviews and related tests, providing support and consultancy in QA matters.

• Allow EMI to pass customer acceptance criteriacontinually improve software quality and process itself

DoW SO 1.5• Continuous building, integration and testing process

by selecting and maintaining tools and resources for building and testing either within the project or in collaboration with external resource providers

EMI I

NFSO

-RI-2

6161

1

5

SA2 Quality Assurance


QA Policies

Build and Test Tools

Integration Infrastructure

QA Metrics

Quality Control

Reports & Dashboards

Development Trackers

NA2JRA1

SA1

JRA1

SA1

SA2Quality Assurance

EMI I

NFSO

-RI-2

6161

1

EMI 2nd EC Review 6

EMI merged 4 established Middleware projects• Each had its own QA practices, tools, reports, etc.

Working with a 3-years vision • Y1: explain, define and implement • Y2: review and automate• Y3: consolidate and optimize

Benefitted from existing QA practices • Used existing QA tools, resources and expertise• Involved PTs, SA1, JRA1 and also explained QA constraints• Associated policies with tools to support them for build, test,

release and automated reports

SA2 QA Strategy

12/06/2012

EMI I

NFSO

-RI-2

6161

1

EMI 2nd EC Review

1. Resubmit SQAP deliverable• Submitted in October 2011. Reorganized as a plan for the 3 years.2. Reduce correlated deliverables • Merged QA and QC deliverable (DSA2.3.3)3.Integrate Quality Control from SA1 and JRA1 into SA2 • Completed by June 2011• Solid QC integration with policies, metrics and reports4. Foster sustainable solutions after EMI • Added standard tools (Mock, Pbuilder)• Compel compliance to established distributions (EPEL, Debian)• Distribution of buildable sources in addition to binaries

Recommendations from 1st Review

12/06/2012 7

EMI I

NFSO

-RI-2

6161

1

8

All SA2 deliverables submitted

• DSA2.1 - Quality Assurance Planrewritten and resubmitted taking into account the review of Y1

• DSA2.2.3 - QA Tools Documentationupdate of previous deliverables DSA2.2.1 and DSA2.2.2

• DSA2.3.3 - Periodic QA and QC Reportsincludes DJRA1.7.3 - Software Development QC Report

SA2 Deliverables


EMI I

NFSO

-RI-2

6161

1

9

Full support Fedora (SL5, SL6), Debian 6 • Standard build tools (Mock and PBuilder) • Standard packaging/repositories (EPEL and APT)Two ways of building• Use ETICS for build and packaging• Give sources to build with Mock/PbuilderAll EMI software builds for EMI platforms• Single project configuration• Used by PTs and for global Nightly Builds• Better workload management• Maven mirror, VM images for developers• Elastic build infrastructure

EMI Tools and ETICS in Y2

EMI 2nd EC Review12/06/2012

0

1000

2000

3000

4000

5000

6000

7000

Montlhy ETICS Builds

DoW SO 1.5 : Continuous integration, building and testing process

Build and Test Tools

EMI I

NFSO

-RI-2

6161

1

10

Fundamental for releases• 3 platforms x Releases x RCs x 32/64 bits • >200 installations of pre-EMI, EMI-1, EMI-2 products• CERN, CESNET, CNAF INFN, DESY, JUELICH, KOSICE, NIIF Everything has been deployed and tested on the QA Testbed• 15 EMI 1 Updates + EMI 2 Release• Provided feedback on documentation, and deployment issues to PTsFundamental for testing, training, demos, monitoring• Adopted common EMI deployment tests, integration tests across products• Used training and demos, supported “EGI TF 2011 best demo” awardCollaboration with EGI is operational and effective• Releases EMI project regularly pass the EGI validation you will see later• Deployed SAM-NAGIOS for automated testing and monitoring

Testing Infrastructure in Y2

12/06/2012

Integration Infrastructure

DoW SO 1.5 : Continuous integration, building and testing process

EMI I

NFSO

-RI-2

6161

1

EMI 2nd EC Review 11

Metrics on code, process, docs• Reaction to RfCs, delays in releases• Compliance to standards and templatesAutomated reports and live dashboards• Focus on metrics immediately useful • RfC management, certification and testingTotal control of Development Process• Status of the products, tests, trackers• Used by QC and Product TeamsFaster and compliant product releases • Submit all QA issues to PTs for prompt action• Products ready for passing the QC tests

QA Metrics and Reports in Y2

12/06/2012

DoW SO 1.4: Common software quality assurance policies and metrics

QA Metrics

Reports & Dashboards

EMI I

NFSO

-RI-2

6161

1


EMI RfC Dashboard

12/06/2012

Data from >30 trackers of all EMI Product Teams, status of EMI developmentAccess to all the RfC in each tracker, search and select, export to Excel

http://emi-rfc.cern.ch

EMI I

NFSO

-RI-2

6161

1

13

EMI Verification Dashboard


Access to all reports of each product, complete view of the productshttp://emi-verification.cern.ch

EMI I

NFSO

-RI-2

6161

1


Software Quality Assurance Plan updated• Reorganised to match your recommendations• Added milestones and targets that were in other documentsAll Product comply with the platforms supported• Multiple platforms support (release, configuration, packaging, doc, etc.)Quality Control monitors compliance to policies• Uses the QA dashboards to view and extract the desired metrics• Metrics automatically extracted to tabular data (e.g. for SA2 and SA1)• Weekly reports and plots from the metrics and constant follow-upQuality Control has impact on work of Product Teams• Products passing are deployed, product not passing are

discussed with the PTs• RfCs are issued and the execute the changes

QA Policies and Control in Y2

12/06/2012

QA Policies

Quality Control

DoW SO 1.4: Common software quality assurance policies and metrics

EMI I

NFSO

-RI-2

6161

1

15

QC of SA2 Services (KPIs)


KPI Target P1 Q5 Q6 Q7 Q8 P2 KSA2.2 - Services Availability

ETICS 97% 97.5 % 98.0 % 95.8 % 96.8 % 97.9 % 97.1 %Testbed 97% 98.3 % 99.0 % 98.9 % 99.1 % 99.3 % 99.1 %

KSA2.3 - Testbed Size 200 CPUs 73 97 112 154 204 204

KSA2.6 - No of RequestsETICS - 230 33 36 47 46 162

Testbed - 35 14 8 11 5 38 KSA2.7 - Average Response Time

ETICS - 6.4 h 3.2 h 2.45 h 0.95 h 0.73 h 1.68 hTestbed - 5.2 h 8.1 h 2.65 h 4.7 h 1.30 h 5.0 h

KSA2.8 - Average Solution TimeETICS - 21.9 h 96.6 h 133.0 h 84.2 h 57.9 h 90.0 h

Testbed - 20.3 h 45.5 h 46.6 h 18.5 h 39.6 h 37.1 h

EMI I

NFSO

-RI-2

6161

1


QC – Testing Improvements

12/06/2012

EMI 1

Updates 1-7 Updates 8-14

Quality Control

EMI I

NFSO

-RI-2

6161

1

17

QC – Regression Testing


Report of Regression Tests for High/Immediate RfCs

Data not collected before

update 5

Policy enforced by Release Manager

after update 7

Quality Control

EMI I

NFSO

-RI-2

6161

1


Convergence to standards of EMI software

QC – Compliance to Fedora/EPEL

12/06/2012

Quality Control

90 %

60 %

EMI I

NFSO

-RI-2

6161

1

19

QC – Defect per Product


Quality Control

Number of Defects per Products per Quarter

EMI I

NFSO

-RI-2

6161

1

20

From EGI TCB 11 – April 2012 DoW SO 1.4: Allow EMI to pass customer acceptance criteria

80%

70%

90%

100%

EMI I

NFSO

-RI-2

6161

1


Main Achievements• New platforms successfully supported• Convergence to standards, QA has an impact on the actual quality• Faster testing and successful deployment of products• Very high success rate in the acceptance of our releases to EGIExcellent collaboration and results across all EMI activities• SA1 for releases, updates and deployment. NA2 for training, documentation.

JRA1 quality control and dashboardsQuality Control merge in SA2 a very good decision• Quicker feedback to EMT and within SA2 (metrics, reports, reviews)•Someone external to PTs checks products (deployment, compliance) Y3: Consolidate and Optimize • Continue to support/improve current tools, testbed, dashboard and reports• QC on sustainability of middleware products beyond the EMI Y3

Conclusions

12/06/2012

EMI I

NFSO

-RI-2

6161

1

22

Thank You

Documents

SA2 – Quality Assurance