Agile Development c/w Continuous

Integration and Testing……………………………………………………….………………………………..................................................................................................

JOHN

PAYNE………………………………………...

SENIOR DATA AND SERVICES DEVELOPER

UK DATA ARCHIVE

UNIVERSITY OF ESSEX………………………………………...

DevCon112 APRIL 2013

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Overview of today’s talk

• Brief background: the UK Data Archive and the UK Data Service

• Historical and Current Structure of ADM

• Why use Agile?

• Why use CIT?

• How we use Agile and CIT together

• Where we are going with Agile and CIT?

• Questions?

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

The UK Data Archive and the UK Data Service

• based at the University of Essex since 1967

• curator of the largest collection of digital data in the social sciences

and humanities in the UK

• see data-archive.ac.uk for more details

• makes these available via the new UK Data Service

• UK Data Service also provides value-added services for UK

Census data, government surveys and beyond

• UK Data Service includes Universities of Essex, Manchester

(Mimas, CCSR), Leeds, Southampton, Edinburgh (Edina) and

University College London

• See ukdataservice.ac.uk for more details

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Historical Structure of ADM

• Until two years ago, Application Development and Maintainance (ADM)

used the waterfall method of application development.

• One developer assigned to work on a product from inception to

implementation across all layers of development.

• This had lead to some of the following all too common issues

• Repetition and duplication of code across projects

• Isolated effort – “Stove Pipes”

• Repetition of data across systems – but not quite!

• Poor quality code and lack of consistency

• Unmaintainable code

• Poor scheduling and long lead times

• Unfocused meandering development cycles

• Building some tables != designing a database

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Current Structure of ADM

Application Development and Maintenance (ADM) now consists of two

teams which opens up Agile possibilities:

• Applications

• Front end development

• Umbraco/MVC/Webforms Java/Solr

/Sharepoint…..

• Services

• Back end development

• C# Web services/Tools/CIT/Databases/Data feeds……

Maintenance functions can fall within either team if appropriate and sensible.

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Why use Agile

• The introduction of Agile techniques has had many benefits:

• Focused project management and decision making during a sprint

• Reduced development time

• Better utilisation of resources and skills

• Technology specialists rather than application specialists

• Improved communication within ADM itself

• Stand up Meetings

• Whiteboards

• Improved communication outside of ADM

• All interested parties encouraged to attend stand ups – not just the team

and the project manager.

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Why use CIT

• To avoid the mistakes and pitfalls of the past and improve turnaround,

quality and reuse.

• With reuse comes dependency – not always visible to a developer

• Small changes can lead to big problems – library reuse

• Errors are spotted early

• Resource changes should not lead to unmaintainable code

• Enforces stylistic rules for readability

• Force developers to produce documentation

• Improve quality and reliability

• Resharper encourages refactoring – BUT handle with care!

• Builds (DLLs) are automatically versioned.

• Essential for managing deployments

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

How we use Agile

• Our main ‘tools’ for Agile are

• Requirements

• User Stories “Given that … when … then ….”

• JIRA – Issue Tracking software

• Time logging and sprint management

• Greenhopper – Plugin to JIRA

• Scrum

• Kanban

• Scrumban!

• Prioritising

• Burndown graphs

• Sprints – One or Two Weeks

• Daily standups

• 15 minutes at 9.30am

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Sprint Planning in JIRA using Greenhopper

Sprint

Commitment

Sprint

Backlog

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

KanBan Board within JIRA

Swimlanes

are

configurable

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Jira Project reporting showing sprint activity

Sprint 1

Planning

Sprint 1

activity

Sprint 1 review

Sprint 2

Planning

Sprint 2

activity

Go Live!

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

How we use CIT

• Our main ‘tools’ for CIT are

• User Stories – Requirements

• JIRA – Issue Tracking software

• SVN – Source Control Repository

• Jenkins – CIT Server

• Resharper – Visual Studio Tool

• Stylecop – Code style ruleset

• Nunit (Rhino Mocks and MOQ)

• Selenium – UI Testing

• Dogygen – API documentation

• Robocopy – Depoloyment

• Sonar – Monitors Coverage

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

How does Jenkins fit in?

Developer PC

• Source Code

• Test Code

• Shared Libraries

• Resharper

• Stylecop

Jenkins

• Build Script

• Stylecop

• Tools

Workspace

• Builds,

• Test results

• Documentation

SVN

• Source Code

• Test Code

• Shared LibrariesCheck in/out

Monitor builds

Build occurs

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Jenkins Screenshots

Job Status

Build History

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Complete CIT flow

Developer PC

Jenkins

SVN

Test Server

Live Server

Staging Server

Monitor builds and failures

Email notifications sent

Code development

Test development

SVN Trigger• Pre-Build – DLL Version

• Build

• Unit Tests

• Selenium Tests

• Code Coverage

• API DocumentationJenkins

Success!

Triggers further

‘lift and shift’ job from

Jenkins’s Workspace.

Updates JIRA issues

Successful build triggers

further jenkins job and

updates JIRA

Deployment script using Robocopy

Jenkins (Manual)Manually triggered

‘lift and shift’ job Jenkins (Manual)Manually triggered

‘lift and shift’ job

fires remote

build

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Agile and CIT Issues

• Because of the team structure within ADM, any application sprint

requires a pre-sprint by the services team.

• How far to go with testing?

• Code Coverage and Mocking

• ‘Functional’ Tests and Unit Tests

• Interfaces!

• Estimating

• Time logging is good

• R&D during a sprint is bad.

• Creep (requirements churn)

• UI developer can end up building to an outdated requirement.

• Communication becomes paramount. ‘Speak up at a stand up’!

• JIRA email blindness

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Where we are going with Agile and CIT

• Database and deployment tools – Redgate Developer Tools

• All new developments involving 3 or more people to be developed as

sprints.

• Thorough testing and continual upskilling

• Test-Driven Development (TDD)

• Behaviour-Driven Development (BDD)

• Improved planning and estimation – Jira time tracking

• Improved products

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Useful Links

• Jira http://www.atlassian.com/software/jira/overview

• Greenhopper http://www.atlassian.com/software/greenhopper/overview

• SVN http://subversion.apache.org/

• Jenkins http://jenkins-ci.org/

• Selenium http://docs.seleniumhq.org/

• Stylecop http://stylecop.codeplex.com/

• Resharper http://www.jetbrains.com/resharper/

• Redgate http://www.red-gate.com/products/sql-development/sql-

developer-bundle/

• Doxygen http://www.stack.nl/~dimitri/doxygen/index.html

• Sonar http://www.sonarsource.org/

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

Thanks for listening

• Any Questions?

……………………………………………………………………………………………………………………………….……………………………..

…………………………………………………………………………………………………………………………………………………………..…

UK DATA ARCHIVE

CONTACT

UK DATA ARCHIVE

UNIVERSIY OF ESSEX

WIVENHOE PARK

COLCHESTER

ESSEX CO4 3SQ

………………………………………

T +44 (0)1206 872001

E info@data-archive.ac.uk

W data-archive.ac.uk