Devops and Programmatic Infrastructure

Devops And Programmatic Infrastructure

William El Kaim Oct. 2016 - V 2.1

This Presentation is part of the

Enterprise Architecture Digital Codex

http://www.eacodex.com/Copyright © William El Kaim 2016 2

http://www.eacodex.com/

Plan

What is Agile Development?

• Agile Development Requires Continuous Delivery

• What is Devops?

• Devops Tools Landscape

• From Devops to Programmatic Infrastructure

Copyright © William El Kaim 2016 3

The Agile Manifesto


Agile Principles (1/2)

• Our highest priority is to satisfy the customer through early and continuous

delivery of valuable software

• Welcome changing requirements, even late in development. Agile processes

harness change for the customer's competitive advantage

• Deliver working software frequently, from a couple of weeks to a couple of

months, with a preference to the shorter timescale

• Business people and developers must work together daily throughout the

project

• Build projects around motivated individuals. Give them the environment and

support they need, and trust them to get the job done

• The most efficient and effective method of conveying information to and

within a development team is face-to-face conversation


Agile Principles (2/2)

• Working software is the primary measure of progress

• Agile processes promote sustainable development. The sponsors,

developers, and users should be able to maintain a constant pace

indefinitely

• Continuous attention to technical excellence and good design enhances

agility

• Simplicity - the art of maximizing the amount of work not done - is essential

• The best architectures, requirements, and designs emerge from self-

organizing teams

• At regular intervals, the team reflects on how to become more effective, then

tunes and adjusts its behavior accordingly


Scrum: Agile Methodology

http://scrumalliance.orgCopyright © William El Kaim 2016 7

http://scrumalliance.org/

Daily Scrum

• Meet every day for 15 minutes

• What did I do yesterday?

• What will I do today?

• What impediments do I encounter?

• Self organize

• Transparency / Visibility

• “ No good or bad news, just transparency ”

Ken Schwaber


Chicken and Pig

• The pigs

• Are “committed”

• Has the right to express their opinion

• Do something for the project

• The chickens

• Are “involved”

• Just listen, don’t speak

• Are interested by the project, but don’t

participate


Pull The Tasks

• Pulling the tasks…

• Is more effective for empirical processes

• Enable team members initiative

• Fits continuous adaptation

• While pushing tasks…

• Is rigid

• Is “control / command”

• Fits defined processes


Scrum Roles


Scrum Features With Use Cases

Coarse grained = Use Cases

Fine grained = Specific scenarios


Scrum Features With User Stories


GUI Low-fidelity Prototypes

Hand-drawn user interface prototype

Post-it-based user interface prototype


Broken Window !

• It leads to software entropy

• Unfixed bugs instill the idea that bug fixing is

unimportant

• Is “technical debts”

• Slow down the development pace

• Lead to “out of control” quality.

• Don’t leave them unrepaired !

• Correcting the situation later is far more

expensive !


At Sprint End: Review Process

• A running product demonstration• Not a slide show

• Execution of business tasks

• Focus on feedback

Product demonstration

Sprint Review meeting

Sprint Retrospective

• An “inspect and adapt” phase for the product owner

• Assessment of the product increment

• What goes right / what goes wrong

• What to do next

• An “inspect and adapt” phase for the team• Improve team practices

• Improve the team efficiency

• Help the team working in better conditions


Sprint retrospectives

Set the stage

Gather data

Generate insights

Decide what to doClose the retrospective

• What are the major events of the Sprint timeline ?

• What have we learned ?

• What worked well and we want to continue ?

• What do we want to do differently ?

• What we want to not repeat ?


Plan

• What is Agile Development?

Agile Development Requires Continuous Delivery

• What is Devops?




Continuous …

• Continuous Integration

• …is a software development practice where engineers integrate frequently, leading to

multiple integrations per day. Each integration is verified by an automated build and test

to detect integration errors as quickly as possible.

• Continuous Delivery

• is a set of practices and principles aimed at Building, Testing, and releasing software

faster and more frequently


Principles for Continuous Integration

• Maintain a Single Source Repository

• Everyone commits to mainline every day

• Every Commit should build the mainline on integration machine

• Keep the Build fast

• Everyone can see what's happening


Continuous Delivery

• “Continuous Delivery is a set of practices and principles aimed at Building,

Testing, and releasing software faster and more frequently”.

• Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment

Automation, by Martin Fowler.

• Key Principles:

• The process for releasing/deploying software MUST be repeatable and reliable.

• Automate everything!

• Done means “released”.

• If something difficult or painful, do it more often.

• Keep everything in source control

• Build quality in!

• Improve continuously.

• Everybody is responsible for release process


http://www.amazon.com/gp/product/0321601912?tag=contindelive-20

Continuous Delivery

Source: http://continuousdelivery.comCopyright © William El Kaim 2016 22

https://blog.codecentric.de/en/2012/08/tutorial-create-a-jenkins-plugin-to-integrate-jenkins-and-nexus-repository/

http://continuousdelivery.com

Continuous Delivery

Source: CollabnetCopyright © William El Kaim 2016 23

https://www.open.collab.net/media/pdfs/2012-renovatingapp.pdf

Continuous Delivery benefits

• Remove waste, risk and bottlenecks in the process to deliver changes faster

and safer

• Improved release quality and predictability

• Improved collaboration

• Test, support, development and operations teams work as one delivery team.

• Predictable and faster turnaround for business requests

• Improve global knowledge of the application

• Improve maturity of the team


Optimizing Development and Operations

Source: HPCopyright © William El Kaim 2016 25

http://fr.slideshare.net/AccentureTechnology/accenture-devops-delivering-applications-at-the-pace-of-business

Deployment Pipeline Process

• Deployment Pipeline models the part of your project’s value stream that goes

from check-in to release, and then to automate it.

• It should be defined based on your teams need and maturity!


Plan



What is Devops?




Symptoms …

• Defects released into production, causing outage

• Inability to diagnose production issues quickly

• Problems appear in some environments only

• Blame shifting/finger pointing

• Long delays while dev, QA, or another team waits on resource or response

from other teams

• “Manual error” is a commonly cited root cause

• Releases slip/fail

• Quality of life issues in IT

Source: TheAgileADmin.comCopyright © William El Kaim 2016 28

https://theagileadmin.com/

Devops Movement

Finishing What Agile Started

• The unrelenting pressure to deliver more application functionality in less time

has given rise to other important trends: Agile Development as a

development methodology and “DevOps” as a methodology for continuously

building and deploying applications in different environments (development,

test, staging, production).

• Agile Development focuses upon having one developer responsible for each component

of an application system, and then having those developers work as a self-coordinating

team to deliver new functionality into production on regular and short time intervals.


Devops Movement

Finishing What Agile Started

Source: APMExpertsCopyright © William El Kaim 2016 30

http://www.ciosummits.com/media/solution_spotlight/AppFirst-WP-Need-For-Application-Operations-in-Dynamic-Data-Center-and-Cloud.pdf

Devops Definitions

• Wikipedia

• DevOps is about eliminating the walls between application development and production

application support, creating one integrated team that builds the application and

supports it in production.

• Forrester

• “A set of processes, methods, and systems for communication, collaboration, and

integration among the IT functions responsible for application development,

infrastructure and operations, and quality assurance; with the functions working together

to produce fit-for-purpose and timely software products and services”.


Devops Definitions

• DevOps refers to a collaborative culture between

technical teams.

• It encompasses much more than just development and

operations

• Extending to QA testing, release, and beyond.

• By aligning goals and sharing strategies, DevOps

practices can improve efficiency and quality of code

shipped, allowing businesses to innovate quickly while

providing a higher standard of support.

• Resources

• http://devops.com/

• http://dev2ops.org/


http://devops.com/

http://dev2ops.org/

Devops History

• ITIL, ITSM, ESM, etc. underdeliver in IT from 1989 on

• Agile comes to the developer world in 2001

• Lean comes to the developer world in 2003 (more slowly)

• O’Reilly Radar “Operations: The New Secret Sauce” in 2006

• Agile Infrastructure discussions start in Europe circa 2007

• Patrick Debois and Andrew Schafer meet up at Agile 2008

• O’Reilly Velocity Conference starts 2008

• Velocity 2009, seminal John Allspaw “10+ Deploys Per Day: Dev and Ops Cooperation” presentation

• Patrick Debois and Kris Buytaert put together first DevOpsDays in Ghent in 2009. Many more follow

• Lean influences enter DevOps via startup culture

• Large companies start branding DevOps “solutions”

Source: TheAgileADmin.comCopyright © William El Kaim 2016 33

https://theagileadmin.com/

Devops Process

Source: MicrosoftCopyright © William El Kaim 2016 34

Plan



• What is Devops?

Devops Tools Landscape


Devops: Most Used Tools


Devops Tools Landscape


Devops Tools LandscapeAnsible vs. Chef vs. Fabric vs. Puppet vs. SaltStack

Copyright © William El Kaim 2016 38Source: IBM

As Oct. 2015

http://fr.slideshare.net/DanielKrook/caps-whats-best-for-deploying-and-managing-openstack-chef-vs-ansible-vs-puppet-vs-salt

Devops Tools Landscape: Ansible

• Ansible is an open source tool used to deploy applications to remote nodes

and provision servers in a repeatable way.

• It gives you a common framework for pushing multi-tier applications and application

artifacts using a push model setup.

• Ansible is built on playbooks that you can apply to an extensive variety of systems for

deploying your app.

• When to use it?

• You do not want to install agents on remote nodes or managed servers.

• Need is to focus on the system administrator side.

• Pricing (subject to change)

• Free open source version,

• Paid plans for Ansible Tower starting at $5,000 per year (which gives you up to 100

nodes).

Source: TaKipi BlogCopyright © William El Kaim 2016 39

http://www.ansible.com/

http://blog.takipi.com/deployment-management-tools-chef-vs-puppet-vs-ansible-vs-saltstack-vs-fabric/

Devops Tools Landscape: Ansible

• Pros• SSH-based, so it doesn’t require installing any agents on remote nodes.

• Easy learning curve thanks to the use of YAML.

• Playbook structure is simple and clearly structured.

• Has a variable registration feature that enables tasks to register variables for later tasks

• Much more streamlined code base than some other tools

• Cons• Less powerful than tools based in other programming languages.

• Does its logic through its DSL, which means checking in on the documentation frequently until you learn it

• Variable registration is required for even basic functionality, which can make easier tasks more complicated

• Introspection is poor. Difficult to see the values of variables within the playbooks

• No consistency between formats of input, output, and config files

• Struggles with performance speed at times.



Devops Tools Landscape: Chef

• Chef is an open source tool for configuration management, focused on the developer side for its user base. • Chef operates as a master-client model, with a separate workstation needed to control

the master.

• It’s based in Ruby, with pure Ruby used for most elements you write.

• When to use it• Before considering Chef, make sure you’re familiar with Git, as it’s required for

configuration, and Ruby, as you’ll have to be writing in it.

• Chef is good for development-focused teams and environments. It’s good for enterprises looking for a more mature solution for a heterogeneous environment.

• Pricing (subject to change)• Free open source version, standard

• Premium plans priced on a per node per month basis that can get down to $6/node/month or $6.75/node/month respectively at high volume.


https://www.chef.io/


Devops Tools Landscape: Chef

• Pros

• Rich collection of modules and configuration recipes.

• Code-driven approach gives you more control and flexibility over your configurations.

• Being centered around Git gives it strong version control capabilities.

• ‘Knife’ tool (which uses SSH for deploying agents from workstation) eases installation

burdens.

• Cons

• Learning curve is steep if you’re not already familiar with Ruby and procedural coding.

• The Chef design is transparent and based on following the instructions it’s given, which

means that you’ll have to make sure your instructions are clear.

• It’s not a simple tool, which can lead to large code bases and complicated environments.

• Doesn’t support push functionality.



Devops Tools Landscape: Fabric

• Fabric is a Python-based tool for streamlining SSH in application

deployments.

• Its primary usage is for running tasks across multiple remote systems, but it can also be

extended with plugins to provide more advanced functionality.

• Fabric will configure your system, do system/server administration, and automate the

deployment of your app.

• When to use it

• If you’re just starting out in the deployment automation space, Fabric is a good beginning

point. It helps if your environment involves at least a little bit of Python.

• Pricing (subject to change)

• Free


http://www.fabfile.org/


Devops Tools Landscape: Fabric

• Pros

• Good at deploying apps written in any language. It doesn’t depend on system

architecture, but rather OS and package manager.

• Simpler and easier to deploy than some other tools in this space

• Extensively integrated with SSH for script-based streamlining

• Cons

• Fabric is a single point of failure set up (generally the machine you’re running the deploy

on)

• Uses a push model, so not as well suited for a continuous deployment model as some

other tools in this space

• While it’s a great tool for deploying apps in most languages, it does require Python to

run, so you must have at least a little Python in your environment for Fabric.



Devops Tools Landscape: Puppet

• Puppet is one of the long standing open source tools in the full-fledged configuration management space.• Puppet is based in Ruby, but uses a customized Domain Scripting Language (DSL)

closer to JSON for working within it.

• It runs as a master-client setup and uses a model-driven approach.

• The Puppet code design works as a list of dependencies, which can make things easier or more confusing, depending on your setup.

• When to use it• Puppet is a good choice if stability and maturity are key factors for you. It’s good for large

enterprises with a heterogeneous environment and range of skills on the DevOps team.

• Pricing (subject to change)• Puppet comes in a free open source version

• Paid commercial enterprise version that runs $112 per node per year, with volume discounts.


https://puppetlabs.com/


Devops Tools Landscape: Puppet

• Pros• Well-established support community through Puppet Labs.

• It has the most mature interface and runs on nearly every OS.

• Simple installation and initial setup.

• Most complete Web UI in this space.

• Strong reporting capabilities.

• Cons• For more advanced tasks, you will need to use the CLI, which is Ruby-based (meaning

you’ll have to understand Ruby).

• Support for pure-Ruby versions (rather than those using Puppet’s customized DSL) is being scaled back.

• Because of the DSL and a design that does not focus on simplicity, the Puppet code base can grow large, unwieldy, and hard to pick up for new people in your organization at higher scale.

• Model-driven approach means less control compared to code-driven approaches.



Devops Tools Landscape: Saltstack

• SaltStack (or Salt) is a CLI-based tool that can be set up as a master-client model or a non-centralized model. • Based in Python, Salt offers a push method and an SSH method of communication with

clients.

• Salt allows for grouping of clients and configuration templates to simplify the control of the environment.

• When to use it• Salt is a good choice if scalability and resiliency are a big concern. It’s good for system

administrators thanks to its usability.

• Pricing (subject to change)• Free open source version

• SaltStack Enterprise version that is based on an annual per node subscription basis. Specific pricing is not listed on their site (just a “Contact us” link), but others have reported a $150 per node per year starting point.


http://saltstack.com/


Devops Tools Landscape: Saltstack

• Pros

• Straightforward organization and usage once you’re past the setup phase.

• Their DSL is feature-rich and isn’t required for logic and states.

• Input, output, and configs are very consistent – all YAML.

• Introspection is very good. It’s easy to see what’s happening within Salt.

• Strong community.

• High scalability and resiliency in the master model with minions and hierarchical tiers.

• Cons

• Difficult to set up and to pick up for new users.

• Documentation is challenging to understand at the introductory level.

• Web UI is newer and less complete than other tool’s Web UIs in the space.

• Not great support for non-Linux OSs.




Copyright © William El Kaim 2016 49Source: IBM

http://fr.slideshare.net/DanielKrook/caps-whats-best-for-deploying-and-managing-openstack-chef-vs-ansible-vs-puppet-vs-salt


• Which configuration management or deployment automation tool you use will

depend on your needs and preferences for your environment.

• Chef and Puppet are some of the older, more established options, making them good for

larger enterprises and environments that value maturity and stability over simplicity.

• Ansible and SaltStack are good options for those looking for fast and simple solutions

while working in environments that don’t need support for quirky features or lots of OSs.

• Fabric is a good tool for smaller environments and those looking for a more low lift and

entry level solution.



Devops Tools Landscape: AWS CloudFormation

• AWS CloudFormation gives developers and systems administrators an easy

way to create and manage a collection of related AWS resources.


Devops Tools Landscape: Terraform

• Infrastructure as Code

• Infrastructure is described using a high-level

configuration syntax. This allows a blueprint

of your datacenter to be versioned and

treated as you would any other code.

Additionally, infrastructure can be shared and

re-used.

• Execution Plans

• Terraform has a "planning" step where it

generates an execution plan.

• Resource Graph

• Terraform builds a graph of all your

resources, and parallelizes the creation and

modification of any non-dependent resources.

• Change Automation

• Complex change sets can be applied to your

infrastructure with minimal human interaction.


Devops Real Examples

• Ansible

• Bower

• Capistrano

• Chef

• Cloud66

• Puppet

• Salstack

• Travis CI / Jenkins CI / Snap-CI

• Vagrant

http://stackshare.io/devopsCopyright © William El Kaim 2016 53

http://www.ansible.com/home

http://bower.io/

http://capistranorb.com/

http://www.cloud66.com/home

https://puppetlabs.com/

https://saltstack.com/

https://travis-ci.org/

https://jenkins-ci.org/

https://snap-ci.com/

https://www.vagrantup.com/

http://stackshare.io/devops

Devops Real Examples

Source: MicrosoftCopyright © William El Kaim 2016 54

Plan



• What is Devops?


From Devops to Programmatic Infrastructure


Rise of Devops

• The rise of DevOps has been part of a three-pronged revolution that is

occurring across industry as businesses grapple with managing new digital

environments:

• An infrastructure revolution is moving much of enterprise to cloud or hybrid cloud

server environments

• An architectural revolution has given rise to APIs, containers and microservices, and

the use of backend-as-a-service products to enable faster creation of single purpose

apps with richer functionality

• An operational revolution has then evolved where these technological revolutions are

changing the way business is organized: business and tech teams are working more

closely together, and every business unit is taking on its IT responsibilities and decision-

making.

56

On The Road To Software-defined Everything

• Amid the fervor surrounding digital, analytics, and cloud, it is easy to

overlook advances currently being made in infrastructure and operations.

• The entire operating environment (server, storage, and network) can now be

virtualized and automated.

• The data center of the future represents the potential for not only lowering

costs, but also dramatically improving speeds and reducing the complexity of

provisioning, deploying, and maintaining technology footprints.

• Software-defined everything can elevate infrastructure investments, from

costly plumbing to competitive differentiators.

Source: DeloitteCopyright © William El Kaim 2016 57

http://www2.deloitte.com/ng/en/pages/technology/articles/tech-trends.html

Cloud Taxonomy

Source: BSA Copyright © William El Kaim 2016 58

http://portal.bsa.org/samcloud/bsasamcloud.pdf

Cloud Taxonomy

Source: BSA Copyright © William El Kaim 2016 59

http://portal.bsa.org/samcloud/bsasamcloud.pdf

CIMI: Cloud IAAS Standard

• Cloud Infrastructure Management Interface

• Specification that standardizes interactions between cloud environments to achieve

interoperable cloud infrastructure management between service providers and their

consumers and developers, enabling users to manage their cloud infrastructure use

easily and without complexity.

• Primer

• Cloud Infrastructure Management Interface Model and RESTful HTTP-based Protocol


http://dmtf.org/sites/default/files/standards/documents/DSP2027_1.0.0.pdf

http://dmtf.org/sites/default/files/standards/documents/DSP0263_1.0.0.pdf

Cloud Brokers

GravitantCloudOrbitCopyright © William El Kaim 2016 61

http://www.gravitant.com/

http://www.cloudorbit.com/

Programmatic Infrastructure

• Developers could now program without taking care of the infrastructure and

the different platforms to deploy on

• AWS Elastic Beanstalk lets you deploy your code in seconds.

• AWS CodeDeploy is a continuous delivery and deployment service

• Even the Developer environment (IDE) is now in the Cloud

• Codenvy simplifies setting up environment and running apps

• And the devops could now program the infrastructure like code

• AWS CloudFormation is like « magic » and lets you manage infrastructure with text files

you could store in configuration management tool

• Devops could also reuse Cloud infrastructure and software bricks and assemble them

like legos (like Amazon AWS) or duplos (like BitNami) depending on their granularity.

• Non functional requirements are now available as services

• Everybody could benefit from a world class infrastructure since day one


https://codenvy.com/

http://aws.amazon.com/fr/cloudformation/

Cloud Cost Advisor

• AWS Simple Monthly Calculator & AWS TCO calculator & AWS Trusted Advisor

• CloudCheckr

• Cloudability

• Cloud Cruiser


http://calculator.s3.amazonaws.com/calc5.html

http://aws.amazon.com/fr/tco-calculator/

https://aws.amazon.com/premiumsupport/trustedadvisor/

http://cloudcheckr.com/

https://cloudability.com/

http://www.cloudcruiser.com/

Codenvy & GitHub: The New Dev Platform


https://enterprise.github.com/aws



https://enterprise.github.com/aws

Infrastructure As Lego: Amazon AWS

Source: Amazon AWSCopyright © William El Kaim 2016 65

http://fr.slideshare.net/AmazonWebServices

BitNami

https://bitnami.com/Copyright © William El Kaim 2016 66

https://bitnami.com/

Infrastructure as Lego

Source: Amazon AWSCopyright © William El Kaim 2016 67

http://fr.slideshare.net/AmazonWebServices

Viadeo: No More Datacenter!

Github

CircleCI

Jenkins

AWS S3

AWS AMIspacker.io

cloudformation

Puppet codebase, CloudFormation templates


Infrastructure As Lego: Google Cloud Platform

http://googlecloudplatform.blogspot.fr/2015/03/deploy-popular-software-packages-using-Cloud-Launcher.html

https://cloud.google.com/actual-cloud/


http://googlecloudplatform.blogspot.fr/2015/03/deploy-popular-software-packages-using-Cloud-Launcher.html

https://cloud.google.com/actual-cloud/

Infrastructure As Lego: Microsoft Azure

http://azure.microsoft.com/en-us/Copyright © William El Kaim 2016 70

http://azure.microsoft.com/en-us/


Infrastructure As Lego: IBM BlueMix

https://console.ng.bluemix.net/http://www-01.ibm.com/software/bluemix/Copyright © William El Kaim 2016 72

https://console.ng.bluemix.net/

http://www-01.ibm.com/software/bluemix/

Twitter

http://www.twitter.com/welkaim

SlideShare

http://www.slideshare.net/welkaim

EA Digital Codex


Linkedin

http://fr.linkedin.com/in/williamelkaim

Claudine O'SullivanCopyright © William El Kaim 2016 73




http://fr.linkedin.com/in/williamelkaim

http://www.claudineosullivan.com/