Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Jason Shawn, Senior Director of DevOps, EllucianJesse Gigler, DevOps Engineer, Ellucian

June 21, 2016

Enterprise DevOps at Scale with AWS

Who are we?

Jesse Gigler Jason Shawn

DevOps Engineer

Sr. Director of DevOps@jsin@jessegigler

Ellucian: Enabling Student Success

How we define DevOps

People working together with a common set of tools & goals to achieve the best customer experience

Did we mention DevOps is culture?

Our DevOps mission statement

Continuous delivery

Source: https://github.com/red-gate/continuous-delivery-periodic-table

Prior to DevOpsMostly lift-and-shift into AWSVery little test coverageSecurity scans ad-hocSparse CI, no real CD processesNew node deployments took man-weeks

Comparison

Current stateRefactoring into cloud-native appsImproved test coverageSecurity scans in DevOps pipeline1500+ Jenkins jobs running daily New node deployments took ~4 hours automated

Our DevOps toolchain

Jenkins – orchestration layer

• Amazon EC2 Plugin allows Jenkins to spin up slaves dynamically as needed.

• Folder per product team. Product teams restricted to their folder based on AD Group.

• CloudBees Folder Plus Plugin allows us to constrain projects/folders to specific slave pools. Separate slave pools for different instance profiles for assume-role access.

Packer to create immutable AMIs

• Immutable AMIs ensure all tools and components are included in the development lifecycle.

• AMIs can be spun up in a different account to audit software and licensing without direct access to the product environment.

• New AMIs are rolled out by updating the launch configuration in terraform.

Terraform for infrastructure as code

• Remote Amazon S3 state-file allows sharing of resource values across modules and teams.

• Count, split, element design pattern to scale resources.

• Jenkins serves as a middleware wrapper to handle dynamic variables and configuration across AWS accounts.

• Allows us to remain “cloud agnostic”.

Ansible for configuration management layer

• Ansible serves as standard format to write and share server-level automation.

• Playbooks are pushed to S3 from Jenkins, then downloaded from S3 and executed in local mode in user_data to provision the instance.

• PowerShell Desired State Configuration for Windows platforms to adhere to the same Ansible principles (push, pull, local).

Automated tests and scans

• Unit, smoke, and functional tests ensure environment is operating as expected.

• ServerSpec tests to validate infrastructure is configured properly.

• Results proxied through bastion server back to Jenkins for reporting and tracking.

Challenges and pain points

Cross-account deployments

AWS resource tag management

• Lots and lots of teams.

• One central “billing” account.

• Defined “required” tags – but how do you enforce this?

Some select child AWS resources do not support a “tag flow down”.

Requirement: Automate a way to flow the resource tags down from the parents to the children.

Answer: Leverage AWS Lambda to regularly scan the environment and copy tags from parent resources to the appropriate child resources.

Resource tag “flow down”

AWS resource soft limitsEach resource class in each region of all of our accounts has a soft limit but:• How can we monitor our soft limits?• How can we automatically request an increase?• How can we ensure production isn’t affected?

AWS scheduling for cost optimizations

"ScheduledActionUp": {  "Type": "AWS::AutoScaling::ScheduledAction",  "Condition": "DEV",  "Properties": {    "AutoScalingGroupName": {    "Ref": ”us-east-1-DevApp"  },  "MaxSize": "1",  "MinSize": "1",  "DesiredCapacity": "1",  "Recurrence": "0 12 * * 1-5"  }},"ScheduledActionDown": {  "Type": "AWS::AutoScaling::ScheduledAction",  "Condition": "DEV",  "Properties": {    "AutoScalingGroupName": {    "Ref": "us-east-1-DevApp"  },  "MaxSize": "0",  "MinSize": "0",  "DesiredCapacity": "0",  "Recurrence": "0 22 * * 2-6"  }}

Emerging patterns

Blue/Green deployment

Self-healing CI/CD environment

• Deployment and configuration of Jenkins pipeline is fully automated.

• Can seamlessly deploy to new regions or recreate an existing environment. EBS volume snapshot is taken and reattached after recreating.

• Self-healing - Jenkins will recreate itself based on certain Amazon CloudWatch alarms.

Events

DevSecOps

Thank you!