JENKINS AS THE TEST REPORTING

FRAMEWORK

Nitin Sharma, SDET @Komli Media

SPEAKER BIO

• Working as SDET with Komli Media

• An automation geek and always try to do innovative

things and blog about it and having 4 years of

experience in Software Industry

WHY TEST REPORTING FRAMEWORKS ?

To help with informed decision, whether to go

ahead with production deployment of the build

To analyze whether sufficient testing being done

What are code changes ?

What are the corresponding automation changes ?

TC Added, Deleted & Modified

What is the test automation run trend ?

Trend on Release basis

REPORTING FRAMEWORK -

REQUIREMENTS

Web based Tool

Cost Effective

Easy to maintain

Integration with TC Execution Frameworks

TEST REPORTING FRAMEWORKS

TestRail – Manage all your testing efforts in a

central place and modern user interface.

Xqual – Provide very fine interface to manage test

reporting framework.

Test Collab – Manage release base trend over UI

SOLUTIONS

• Continuous Integration

server that allows to run

multiple automation suites

at the same.

• Supports SVN, Git and

about any other SCM

tools you can think of via

plugins.

WHY JENKINS

Web based

No additional cost, we are already dependent on it

as TC Execution framework

Well supported

Highly customizable

JENKINS ON THE BASIS OF RELEASES DONE

COMMON JENKIN TERMS

Project

Builds

NEW TERMS INTRODUCED

Benchmark Project

Benchmark Builds

BENCHMARK PROJECT

The project in Jenkins where we maintain the

automation results on the basis of releases done to

production

BENCHMARK BUILD

Build which has best results

The one whose final status we want to maintain in

the benchmark project

WHAT IS ALREADY AVAILABLE ?

Show automation run trend

Integration with SCM tool show code diffs

CHALLENGES

Integration : Want every thing in the single screen

(single project in Jenkins)

Benchmark Build : We don’t know priori that given

build is the best build to run

Don’t want to waste resources & time to execute

automation run again once we have decided on the

build to deploy on production

Intermittent failures : Want to maintain their status as

Passed for a given release.

If we have taken the approach of copying results from

one project to other project (benchmark project)

OUR APPROACH

Git Plugin for code diff

Custom Java Script for TC diff

Custom Java Script for override intermittent failure

status

CopyArtifact plugin to migrate results

Custom Groovy script for re-generating the Test NG

result trends

INTERMITTENT FAILURES

PARAMETERS TO COPY BENCHMARK BUILD

OVERRIDE INTERMITTENT FAILURE STATUS TO

PASS

FUTURE TASKS

Show modified test-cases with the releases

Show release trend graph in different color for

manually override test-cases

Capturing results of multiple projects in same

dashboard

THANK YOU !!