Ian Watson - DevOps at Callcredit….a year nearer awesome

DevOps at Callcredit

…..a year nearer awesome

Ian WatsonHead of DevOps at CallcreditEmail : [email protected] : @purplemarauder

I am….

mailto:[email protected]

What’s the challenge?

Where we were….

Actual Deployment planCheck if access to the deployment packages is available.

• R:\Callcredit Development DML\LL\Builds\Release- v6.1.56.14856.zipExtract

the relevant (Web or App) files zip within the release to c:\program files\callcredit\LL\ on each Web and App servers. Ensure the correct rollback version location is known.

Deployment Steps

Instruction Done (1,2&3) Done (4,5&6)Run script to take the first 3 pairs of Web and App servers out of the pool“C:\scripts\scripts” on WIN**WB01 server….(run Phase 1 script).

ENSURE YOU ARE NOT REMOVING ALL SERVERS FROM THE POOL

On the app server:

• Open a command prompt in the "Release" folder, e.g. "C:\Program Files\callcredit\LL\ v6.1.56.14856\Release"• Enter the command:• Teardown_APP On the app server:

• Open a command prompt in the "Release" folder, e.g. "C:\Program Files\callcredit\LL\ v6.1.56.14856\Release"• Enter the command:• nant -D:"config=siteconfigid" -D:"encrypt.key=yourencryptionkey“

ptionkey = see app support passwordsafe) (where siteconfigid:• Live = “LV”• Site 2 QA = “qa”• Site 2 CT = “ct”

On the web server, open INETMGR and delete the following:

From IIS ? Callcredit Web Site ? services

• Admin • Call• DL1• DL2• LL

From IIS ? Callcredit Web Site

• Call • DL• LL

On the web server:

• Open a command prompt in the "Release" folder, e.g. "C:\Program Files\callcredit\LL\ v6.1.56.14856\Release"• Enter the command:• Teardown_WEB

On the web server:

• Open a command prompt in the "Release" folder, e.g. "C:\Program Files\LL\ v6.1.56.14856\Release"• Enter the command:• nant -D:"config=siteconfigid" -D:"encrypt.key=yourencryptionkey"

(where yourencryptionkey = see app support passwordsafe) (where siteconfigid:• Live = “LV”• Site 2 QA = “qa”• Site 2 CT = “ct”

On each of the App servers:

Under component services\computers\COM+Applications\CAST Job controller\Roles\Permitted Users\

Add the user: ServerName\_LL_CS_WebSvc

e.g. OLSWINxxAP0x\ LL_CS_WebSvc

the user should already be present under computer Management users

On each of the App servers:Open DCOMCNFG -> services\computers\COM+Applications\CAST Job controller

Right click on the component and:

• Shut down• Disable• Enable

DBA should run following script on ML_Lookup_Vws5 databse:

UPDATE vw_IDType SET [Desc] = 'Passport' WHERE [ID] = 5

Rerun the pairing scripts to update COM+ configs (if COM+ has been updated) ENSURE YOU ARE RUNNING THE CORRECT “PHASE” SCRIPT TO MAINTAIN SEPERATION BETWEEN UPGRADED AND NOT UPGRADED VERSIONS OF THE RELEASE!!

QA can then verify the web and app tier install

Add the 3 pairs back into the pool and remove next 3 pairs (run phase 2 script).

ENSURE YOU ARE RUNNING THE CORRECT “PHASE” SCRIPT

Repeat above steps for each pair

Once paired testing is complete, add all servers back into pool (run Phase 3 script)

ENSURE YOU ARE RUNNING THE CORRECT “PHASE” SCRIPT

QA will perform a final BIGIP test.

We needed to change

Research, research, research

What is DevOps?

Universal Truths

You can’t ‘do’ DevOps…

…. in a windows Environment

…. on an Enterprise scale

…. in a database-centric companyWRONG!

Topologies

The Teams – System Build

The Teams – Platform Build

Click icon to add picture

Death to Manual Deployments!

Numbers from : Octopus Deployment Automation Survey 2013, State of Software Delivery: Trends in the market now and in the future (2014), Puppet Labs State of DevOps Report 2014





Deployment automation - Picking a tool

https://xebialabs.com/periodic-table-of-devops-tools/

Pilot teams

Continuous Delivery

What :

We make sure our software is always potentially shippable throughout its entire lifecycle and that any build could potentially be released to users at the touch of a button using a fully automated process.

Why :

We can put the release schedule in the hands of the business rather than IS. Be this with a regular rhythm or on demand.

Deploying Databases isn’t easy

But it doesn’t have to be difficult either

Aug Sept Oct Nov Dec Jan Feb Mar Apr0

10

20

30

40

50

60

70

80

Test Pre-Prod Prod

Click icon to add pictureDeployments/Month

Meanwhile – Platform & Infrastructure….

Tooling, tooling, tooling

Achievements so far

Achievements so far…

People actually understand what DevOps is!

NOT A TEAM!!

In fact…

We now have an active DevOps Community


A new definition of done

“it’s been coded and tested”

“it’s working in Production”


Packaged, scripted deployments are now the standard rather than the exception

40% of products deploying via Octopus Deploy


Able to build dozens of version controlled servers in very short timescales

Not all smooth sailing

Culture change is HARD!

Technical challenges

Too fast?

Still plenty of work to do to get to….

Next steps along the road - Culture- Complete the move from Project Teams to Product Teams- Really treating Pre-Production and Production the same

- We need to breed more champions across the company – Development, Security, Operations, Business

Next steps along the road - Monitoring- MUCH more monitoring – proactive rather than reactive and visible to everyone- Log aggregation – choose a tool and allow access to Dev and Ops- Get development teams delivering monitoring as a standard part of any system

Next steps along the road - Environments- IaaS – internally and externally

- MUCH more monitoring – proactive rather than reactive and visible to everyone

- The Cloud – use the best technology for the required solution

Next steps along the road – Delivery- Blue/Green deployments with zero downtime- Continuous performance testing

Anyone can ‘do’ DevOps

If we can…..

Just keep aiming at Awesome!

Leadership & Management

Ian Watson - DevOps at Callcredit….a year nearer awesome