Octopus Deploy Customization and Automation

with .NET and PowerShellThere and back again

Who am I?

Ian Paullin Architect/DevOps/.NET Programmer 5 years with Avanade, 10 years in .NET New Dad Seahawks Fan

What are we talking about?

Our story with Octopus Intro to Octopus REST API How to get started Demos

Octo.exe PowerShell Demo Post message to Slack

Why are we talking about this?

Octopus is awesome Automation is awesome Octopus is extremely extensible Octopus extensibility + automation = more awesome

I thought it would be like..

In retrospect..

Our Story

Take Octopus Deploy to the masses. Ensure 24/7 uptime. Limit specific users to deploy to production.

February 2014 – May 2014 (Pilot approx. 10 teams) June 2014 – February 2015

Goal - 120 projects on board by Feb/Mar 2015 February 2015 upgrade: from 2.5.4 to 2.5.12 (eventually to 2.5.13)

Problems emerged with upgrade March 2015 – October 2015

Stabilization + growth November 2015 – January 2016

More stabilization and automation More users/projects/servers Testing Octopus 3.1.x – 3.2.x

Where are we today?

5 Octopus servers with 5 NuGet servers Total tentacles: 1564 Total # of project groups: 255 Total # of projects: 509 Busiest Day: Wednesday – avg. 105 deployments Weekly deployment avg: 300-450 NuGet packages: A lot (e.g. 200Gb on just one of our NuGet servers)

Scaling

Octopus can do it, but we didn’t realize all contributing factors NuGet issues – with free NuGet.Server

Was recommended to use NuGet.Lucene or NuGet Gallery MyGet.org/ProGet No feed maintenance means large feeds over time (3-10gb and thousands of packages) Solution

Reduce packages in feeds (manually )) IIS Fileshares circumvent issue with NuGet client DLL

Octopus Server File Handles Max capacity of listening tentacles (listening uses 1 thread per machine!) NuGet communication issues Solution

Determine upper limit Balance team tentacle size across number of servers Switch to polling mode for tentacles (no thread per each polling tentacle like listening mode) Deleting old/stale projects

Lessons Learned

Research was difficult (at the time) No real documentation on Octopus on massive scaling (everything is relative) My failure to reach out before hand to determine how to best scale Our use case maybe outside of scope of Octopus? Not quite – but 3.x changes everything

about Octopus Deploy Estimation

Pilot proved it could work but estimating scaling or load was very obfuscated by numerous aspects (NuGet, tentacle communications, server configuration, etc.)

Initial estimate of teams moving to Octopus was way too low. Monitoring

SumoLogic – we had for IIS for NuGet.Server but didn’t use it correctly at beginning AppDynamics – added late to really foresee issues but invaluable today Not constantly watching environments – too many machines added

Lessons Learned (con’t)

RavenDB limitations of scale Primarily licensing issue limiting to 3 CPU cores and 6GB (embedded/ISV)

NancyFX is nice but.. Shielded information about user sessions from us Now Octopus 3.x has logging feature for sessions

Understand Limits/Tolerance If you don’t know them, find them out (break your stuff if you can) If you can’t find them, constantly watch out for multiple factors (monitoring) Monitoring especially at enterprise scale is not optional

What really worked well

Provisioning Automate process to create a team within Octopus. By hand, could take 25 to 30 minutes –

sometimes a lot longer. Automating process meant 30 to 45 seconds of filling out fields and executing a deployment

Automatically create NuGet feed on NuGet server for each project (isolation) Integration with Active Directory Groups and TFS AD Groups makes life much easier Can build a customize new Octopus Deploy server with one provisioning script

Permissions Permissions model is complicated, but extremely effective for our needs. This is 100%

automated from the provisioning process Reporting

Reports on how many deployments; how frequent; to which environments, etc. Error reports and frequency tracking

What really worked well (con’t)

Synchronizing user accounts with internal service Get list of users that are approved for production deployments for specific security teams – checks

every hour across all our instances Monitoring

AppDynamics – late start but very helpful in keeping a watch on stability Server metrics to gauge stability and measure improvements

Maintenance Weekly reboots – fully automated NuGet cleanup – not automated (yet) Changing IIS feeds to filesharing

Constant improvements Weigh pros/cons, time available and payoff Target daily or common tasks/procedures Backlog one-offs, random issues, odds/ends

Odds and ends

User Education Some of our users still don’t use optimize Octopus. They can cause lots of

problems with an entire instance and can make server unstable at least for 2.5.13 (backups, canceling/starting deployments, long running processes, etc.)

Documentation + videos + guidance sessions still not enough for some Good users are always willing to learn and improve but it takes time to see

benefits Excellent users are fewer in number, but really good in maximizing octopus Teams change therefore users change; re-education is cyclical and therefore

problematic over time

Support directly from Octopus Deploy Great support from small company Weekly meetings, research and testing extremely helpful Helping us now with our plans to migrate to 3.1.x – 3.2.x Continuing guidance and support on architecture design and

migration to 3.x

Keys to winning

Perseverance Keep at it

Watch closely and at high-level Monitoring + reports help make sense of what’s going on and what may

happen Continually improve aspects in and around Octopus Deploy

Improvements or tweaks, no matter how small, add value over time Automate

With Octopus REST API, we can script repeatable tasks for everything Customize

With Octopus REST API, we can extend Octopus in almost any way we want

One key to winning: The Octopus REST API View the API at http://<your octopus server>/api Documentation:

http://docs.octopusdeploy.com/display/OD/Octopus+REST+API https://github.com/OctopusDeploy/OctopusDeploy-Api/wiki http://docs.octopusdeploy.com/display/OD/Octopus.Client

Octopus REST API

Everything flows through the REST API Internally Externally Octo.exe

Octopus REST API - Demos

Demos Look at the octo.exe Query Octopus Deploy server directly via PowerShell Simple test message to Slack during deployment

Octopus REST API – DiY with Octo.exe Octo.exe

Great for automating deployments after build Quick Demo with Octo.exe

Octopus REST API – DiY with PowerShell/C# You need 3 DLLs (found in tentacle/server installation folder)

Newtonsoft.Json.dll Octopus.Client.dll Octopus.Platform.dll REST API URL exposes what’s available

APIKey and appropriate permissions for the user account (or service account) dictate what’s allowed for interaction; no username/password!

Quick Demo for PowerShell Report example

Octopus REST API – Sample Report

Ideas Any task done more than two or

three times Windows Tasks that can be

scheduled indefinitely Write to log or database for

auditing history Integration with 3rd party

tools/platforms

Examples Synchronization of users – if a

small subset of users is allowed Scheduled reboots – if necessary Security checks – especially for

AD groups Integration with Slack for

messages of deployments

Automation/Customization Ideas

Integration with Slack

Slack is a 3rd party collaboration/messaging platform that allows users to chat as well as programmable bots or external parties

Idea: for every deployment in Octopus, send message to Slack Demo

Octopus REST API conclusion

Not enough to just automate deployments. There’s lots of things around deployments that need automation as well.

Without the REST API, it’s much harder if not impossible to customize and automate as we did.

Many creative solutions possible – imagination and time are only barriers.

Thanks to..

Octopus Deploy team Damian Brady Vanessa Love Paul Stovell

Thadd Parker (@tptechgeek) For stats and sample report and sample C# code

Gandalf the Pug https://www.reddit.com/r/pics/comments/3pgtyr/gandalf_the_pug/