Coordinating Change and Release Management in

Games DevelopmentGordon Milligan

QA and Release ManagerRealtime Worlds

Intro

• Change control from games perspective• Case study discussing our initial forays

into change control for a large team• Discuss what we learned• A further case study on how we work now

Who are we?

• Software technology company in entertainment sector

• Based in Dundee, Scotland• Largest independent games company in

Scotland–Approximately 300 staff

Change Control

“process of managing, and controlling, how a product is changed”

–Michael E. Bays

Change Control Considerations

• Build changes • Test changes• Review changes

–Quality–Risk

Maintaining Quality: Stability

• Continuous Integration• Daily smoke test at a minimum• Full regression passes• Targeted feature testing• Automated Testing

Maintaining Quality: Overall

• Continuous Review–Designers–Artists–Producers–Creative directors

Risk

• Analyse risk of changes and likely impact–Highlight concerns to developers,

producers, etc.• Attempt to make developers aware of the

risk of their changes

Evolution of Change Processes

• Initially we were naive–Lack of understanding of the issue–Size of team was bigger than any of us

had worked on–Hit many problems: no real control

• Educated ourselves and continued to review and evolve or change processes

•

Development Team Make Up

• ~55 % Software Engineers• ~20 % Artists• ~20 % Level/Script Designers• ~5% Production

Daily Build is Life Blood

• ...for non-programmers• Relied on to allow artists/designers to

receive new code, design & art content• Allows non-coders to develop their content

against recent changes• Provides buffer against bleeding edge

Crackdown Approach

Compunding Problems

• Lack of due diligence on check-ins• Long build verification turnaround• 70 people working on same codeline ==

continuous churn–Constant changes to common files–Productivity loss through broken builds

Process Changes

• Introduced stringent check-in process–Could take 2-3 hours all in (or much

longer on occasion)

Example Check-in Process

• Build data & code• Blow up car• Drive car • Kill NPC• Complete mission X• Etc.

Process Changes

• Introduced stringent check-in process–Could take 2-3 hours all in (or much

longer on occasion)• Analysed and improved build verification

speed on build servers

Reaction to Changes

• Less check-ins so code would change underneath feature development–Also code/data sat on dev machines for

longer so issues hidden• Friday afternoon became favoured check-

in time• Coders loathed submitting changes

Further Process Changes

• Restricted times when check-ins could happen

• Created check-in queue (FIFO)

Problems

• Change submission bottleneck–Whose next in queue?

• Coders not checking in regularly enough so always seeing BIG CUMULATIVE changes

• Major breakages of build and resultant loss of prodctivity

Crackdown: Lessons Learned

• Say NO to...–Large # devs submitting change to same

location–Long build verification cycles (if

possible)–Allowing developers to check-in code

without review

Main thing I learned...

Trade OffMake check-ins easy and less restrictive

VMaintain a stable product at all times

Project X

• Consists of multiple SCRUM teams working on separate areas

• Mixture of industry veterans through to new graduates

• Have a measured approach to quality• Similar team make-up to previous project

Development Process Alterations

• Introduced buddy check-in process to share knowledge/responsibility & reduce risk

• Use unit, integration and higher level automated testing– In addition to black box testing

• Code leads (& QA) meet most mornings

Development Process Alterations

• Mainline is no longer a development codeline and is owned by QA–Used to consolidate features

• SCRUM teams each have own development codeline which they own–Allows bugs to be found early and code

to mature

Development Process Challenges

• Educate developers on:– Benefits of branching

• In particular task & prototype branches to reduce risk

–How to use version control feature set• Encourage regular short check-ins

Benefits of Current Setup

• Lightweight check-in process for team branches–Allows quick change submission–Changes submitted earlier and so have

time to mature and for issues to be found

• SCRUM Masters own team branches

Benefits of Current Setup

• Strict policy for mainline integration helps to ensure its quality–Rarely is it broken

• I have closer control of changes hitting mainline–More time to review these and assess

risk

Hurdles crossed during changes

• Resistance to methodology change–Stakeholder buy in

• Educating developers on:–Branching: when to, how to–Using version control system features

• Communication

Summary

• Discussed types of changes & dependencies involved in game builds

• Talked in detail about the problems of trying to over control change

• Put forward a new strategy we have developed to control change

Tools

• Perforce • CI System

–CruiseControl.net–Moving to Team City

• Jira: Feature tracking (with help of p4 jobs)

Resources

• Software Configuration Management Patterns–Berczuk and Appleton

• Software Release Methodology –Michael E. Bays

• Practical Perforce–Laura Wingerd