Devops And Programmatic Infrastructure
William El Kaim Oct. 2016 - V 2.1
This Presentation is part of the
Enterprise Architecture Digital Codex
http://www.eacodex.com/Copyright © William El Kaim 2016 2
Plan
What is Agile Development?
• Agile Development Requires Continuous Delivery
• What is Devops?
• Devops Tools Landscape
• From Devops to Programmatic Infrastructure
Copyright © William El Kaim 2016 3
The Agile Manifesto
Copyright © William El Kaim 2016 4
Agile Principles (1/2)
• Our highest priority is to satisfy the customer through early and continuous
delivery of valuable software
• Welcome changing requirements, even late in development. Agile processes
harness change for the customer's competitive advantage
• Deliver working software frequently, from a couple of weeks to a couple of
months, with a preference to the shorter timescale
• Business people and developers must work together daily throughout the
project
• Build projects around motivated individuals. Give them the environment and
support they need, and trust them to get the job done
• The most efficient and effective method of conveying information to and
within a development team is face-to-face conversation
Copyright © William El Kaim 2016 5
Agile Principles (2/2)
• Working software is the primary measure of progress
• Agile processes promote sustainable development. The sponsors,
developers, and users should be able to maintain a constant pace
indefinitely
• Continuous attention to technical excellence and good design enhances
agility
• Simplicity - the art of maximizing the amount of work not done - is essential
• The best architectures, requirements, and designs emerge from self-
organizing teams
• At regular intervals, the team reflects on how to become more effective, then
tunes and adjusts its behavior accordingly
Copyright © William El Kaim 2016 6
Scrum: Agile Methodology
http://scrumalliance.orgCopyright © William El Kaim 2016 7
Daily Scrum
• Meet every day for 15 minutes
• What did I do yesterday?
• What will I do today?
• What impediments do I encounter?
• Self organize
• Transparency / Visibility
• “ No good or bad news, just transparency ”
Ken Schwaber
Copyright © William El Kaim 2016 8
Chicken and Pig
• The pigs
• Are “committed”
• Has the right to express their opinion
• Do something for the project
• The chickens
• Are “involved”
• Just listen, don’t speak
• Are interested by the project, but don’t
participate
Copyright © William El Kaim 2016 9
Pull The Tasks
• Pulling the tasks…
• Is more effective for empirical processes
• Enable team members initiative
• Fits continuous adaptation
• While pushing tasks…
• Is rigid
• Is “control / command”
• Fits defined processes
Copyright © William El Kaim 2016 10
Scrum Roles
Copyright © William El Kaim 2016 11
Scrum Features With Use Cases
Coarse grained = Use Cases
Fine grained = Specific scenarios
Copyright © William El Kaim 2016 12
Scrum Features With User Stories
Copyright © William El Kaim 2016 13
GUI Low-fidelity Prototypes
Hand-drawn user interface prototype
Post-it-based user interface prototype
Copyright © William El Kaim 2016 14
Broken Window !
• It leads to software entropy
• Unfixed bugs instill the idea that bug fixing is
unimportant
• Is “technical debts”
• Slow down the development pace
• Lead to “out of control” quality.
• Don’t leave them unrepaired !
• Correcting the situation later is far more
expensive !
Copyright © William El Kaim 2016 15
At Sprint End: Review Process
• A running product demonstration• Not a slide show
• Execution of business tasks
• Focus on feedback
Product demonstration
Sprint Review meeting
Sprint Retrospective
• An “inspect and adapt” phase for the product owner
• Assessment of the product increment
• What goes right / what goes wrong
• What to do next
• An “inspect and adapt” phase for the team• Improve team practices
• Improve the team efficiency
• Help the team working in better conditions
Copyright © William El Kaim 2016 16
Sprint retrospectives
Set the stage
Gather data
Generate insights
Decide what to doClose the retrospective
• What are the major events of the Sprint timeline ?
• What have we learned ?
• What worked well and we want to continue ?
• What do we want to do differently ?
• What we want to not repeat ?
Copyright © William El Kaim 2016 17
Plan
• What is Agile Development?
Agile Development Requires Continuous Delivery
• What is Devops?
• Devops Tools Landscape
• From Devops to Programmatic Infrastructure
Copyright © William El Kaim 2016 18
Continuous …
• Continuous Integration
• …is a software development practice where engineers integrate frequently, leading to
multiple integrations per day. Each integration is verified by an automated build and test
to detect integration errors as quickly as possible.
• Continuous Delivery
• is a set of practices and principles aimed at Building, Testing, and releasing software
faster and more frequently
Copyright © William El Kaim 2016 19
Principles for Continuous Integration
• Maintain a Single Source Repository
• Everyone commits to mainline every day
• Every Commit should build the mainline on integration machine
• Keep the Build fast
• Everyone can see what's happening
Copyright © William El Kaim 2016 20
Continuous Delivery
• “Continuous Delivery is a set of practices and principles aimed at Building,
Testing, and releasing software faster and more frequently”.
• Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment
Automation, by Martin Fowler.
• Key Principles:
• The process for releasing/deploying software MUST be repeatable and reliable.
• Automate everything!
• Done means “released”.
• If something difficult or painful, do it more often.
• Keep everything in source control
• Build quality in!
• Improve continuously.
• Everybody is responsible for release process
Copyright © William El Kaim 2016 21
Continuous Delivery
Source: http://continuousdelivery.comCopyright © William El Kaim 2016 22
Continuous Delivery
Source: CollabnetCopyright © William El Kaim 2016 23
Continuous Delivery benefits
• Remove waste, risk and bottlenecks in the process to deliver changes faster
and safer
• Improved release quality and predictability
• Improved collaboration
• Test, support, development and operations teams work as one delivery team.
• Predictable and faster turnaround for business requests
• Improve global knowledge of the application
• Improve maturity of the team
Copyright © William El Kaim 2016 24
Optimizing Development and Operations
Source: HPCopyright © William El Kaim 2016 25
Deployment Pipeline Process
• Deployment Pipeline models the part of your project’s value stream that goes
from check-in to release, and then to automate it.
• It should be defined based on your teams need and maturity!
Copyright © William El Kaim 2016 26
Plan
• What is Agile Development?
• Agile Development Requires Continuous Delivery
What is Devops?
• Devops Tools Landscape
• From Devops to Programmatic Infrastructure
Copyright © William El Kaim 2016 27
Symptoms …
• Defects released into production, causing outage
• Inability to diagnose production issues quickly
• Problems appear in some environments only
• Blame shifting/finger pointing
• Long delays while dev, QA, or another team waits on resource or response
from other teams
• “Manual error” is a commonly cited root cause
• Releases slip/fail
• Quality of life issues in IT
Source: TheAgileADmin.comCopyright © William El Kaim 2016 28
Devops Movement
Finishing What Agile Started
• The unrelenting pressure to deliver more application functionality in less time
has given rise to other important trends: Agile Development as a
development methodology and “DevOps” as a methodology for continuously
building and deploying applications in different environments (development,
test, staging, production).
• Agile Development focuses upon having one developer responsible for each component
of an application system, and then having those developers work as a self-coordinating
team to deliver new functionality into production on regular and short time intervals.
Copyright © William El Kaim 2016 29
Devops Movement
Finishing What Agile Started
Source: APMExpertsCopyright © William El Kaim 2016 30
Devops Definitions
• Wikipedia
• DevOps is about eliminating the walls between application development and production
application support, creating one integrated team that builds the application and
supports it in production.
• Forrester
• “A set of processes, methods, and systems for communication, collaboration, and
integration among the IT functions responsible for application development,
infrastructure and operations, and quality assurance; with the functions working together
to produce fit-for-purpose and timely software products and services”.
Copyright © William El Kaim 2016 31
Devops Definitions
• DevOps refers to a collaborative culture between
technical teams.
• It encompasses much more than just development and
operations
• Extending to QA testing, release, and beyond.
• By aligning goals and sharing strategies, DevOps
practices can improve efficiency and quality of code
shipped, allowing businesses to innovate quickly while
providing a higher standard of support.
• Resources
• http://devops.com/
• http://dev2ops.org/
Copyright © William El Kaim 2016 32
Devops History
• ITIL, ITSM, ESM, etc. underdeliver in IT from 1989 on
• Agile comes to the developer world in 2001
• Lean comes to the developer world in 2003 (more slowly)
• O’Reilly Radar “Operations: The New Secret Sauce” in 2006
• Agile Infrastructure discussions start in Europe circa 2007
• Patrick Debois and Andrew Schafer meet up at Agile 2008
• O’Reilly Velocity Conference starts 2008
• Velocity 2009, seminal John Allspaw “10+ Deploys Per Day: Dev and Ops Cooperation” presentation
• Patrick Debois and Kris Buytaert put together first DevOpsDays in Ghent in 2009. Many more follow
• Lean influences enter DevOps via startup culture
• Large companies start branding DevOps “solutions”
Source: TheAgileADmin.comCopyright © William El Kaim 2016 33
Devops Process
Source: MicrosoftCopyright © William El Kaim 2016 34
Plan
• What is Agile Development?
• Agile Development Requires Continuous Delivery
• What is Devops?
Devops Tools Landscape
Copyright © William El Kaim 2016 35
Devops: Most Used Tools
Copyright © William El Kaim 2016 36
Devops Tools Landscape
Copyright © William El Kaim 2016 37
Devops Tools LandscapeAnsible vs. Chef vs. Fabric vs. Puppet vs. SaltStack
Copyright © William El Kaim 2016 38Source: IBM
As Oct. 2015
Devops Tools Landscape: Ansible
• Ansible is an open source tool used to deploy applications to remote nodes
and provision servers in a repeatable way.
• It gives you a common framework for pushing multi-tier applications and application
artifacts using a push model setup.
• Ansible is built on playbooks that you can apply to an extensive variety of systems for
deploying your app.
• When to use it?
• You do not want to install agents on remote nodes or managed servers.
• Need is to focus on the system administrator side.
• Pricing (subject to change)
• Free open source version,
• Paid plans for Ansible Tower starting at $5,000 per year (which gives you up to 100
nodes).
Source: TaKipi BlogCopyright © William El Kaim 2016 39
Devops Tools Landscape: Ansible
• Pros• SSH-based, so it doesn’t require installing any agents on remote nodes.
• Easy learning curve thanks to the use of YAML.
• Playbook structure is simple and clearly structured.
• Has a variable registration feature that enables tasks to register variables for later tasks
• Much more streamlined code base than some other tools
• Cons• Less powerful than tools based in other programming languages.
• Does its logic through its DSL, which means checking in on the documentation frequently until you learn it
• Variable registration is required for even basic functionality, which can make easier tasks more complicated
• Introspection is poor. Difficult to see the values of variables within the playbooks
• No consistency between formats of input, output, and config files
• Struggles with performance speed at times.
Source: TaKipi BlogCopyright © William El Kaim 2016 40
Devops Tools Landscape: Chef
• Chef is an open source tool for configuration management, focused on the developer side for its user base. • Chef operates as a master-client model, with a separate workstation needed to control
the master.
• It’s based in Ruby, with pure Ruby used for most elements you write.
• When to use it• Before considering Chef, make sure you’re familiar with Git, as it’s required for
configuration, and Ruby, as you’ll have to be writing in it.
• Chef is good for development-focused teams and environments. It’s good for enterprises looking for a more mature solution for a heterogeneous environment.
• Pricing (subject to change)• Free open source version, standard
• Premium plans priced on a per node per month basis that can get down to $6/node/month or $6.75/node/month respectively at high volume.
Source: TaKipi BlogCopyright © William El Kaim 2016 41
Devops Tools Landscape: Chef
• Pros
• Rich collection of modules and configuration recipes.
• Code-driven approach gives you more control and flexibility over your configurations.
• Being centered around Git gives it strong version control capabilities.
• ‘Knife’ tool (which uses SSH for deploying agents from workstation) eases installation
burdens.
• Cons
• Learning curve is steep if you’re not already familiar with Ruby and procedural coding.
• The Chef design is transparent and based on following the instructions it’s given, which
means that you’ll have to make sure your instructions are clear.
• It’s not a simple tool, which can lead to large code bases and complicated environments.
• Doesn’t support push functionality.
Source: TaKipi BlogCopyright © William El Kaim 2016 42
Devops Tools Landscape: Fabric
• Fabric is a Python-based tool for streamlining SSH in application
deployments.
• Its primary usage is for running tasks across multiple remote systems, but it can also be
extended with plugins to provide more advanced functionality.
• Fabric will configure your system, do system/server administration, and automate the
deployment of your app.
• When to use it
• If you’re just starting out in the deployment automation space, Fabric is a good beginning
point. It helps if your environment involves at least a little bit of Python.
• Pricing (subject to change)
• Free
Source: TaKipi BlogCopyright © William El Kaim 2016 43
Devops Tools Landscape: Fabric
• Pros
• Good at deploying apps written in any language. It doesn’t depend on system
architecture, but rather OS and package manager.
• Simpler and easier to deploy than some other tools in this space
• Extensively integrated with SSH for script-based streamlining
• Cons
• Fabric is a single point of failure set up (generally the machine you’re running the deploy
on)
• Uses a push model, so not as well suited for a continuous deployment model as some
other tools in this space
• While it’s a great tool for deploying apps in most languages, it does require Python to
run, so you must have at least a little Python in your environment for Fabric.
Source: TaKipi BlogCopyright © William El Kaim 2016 44
Devops Tools Landscape: Puppet
• Puppet is one of the long standing open source tools in the full-fledged configuration management space.• Puppet is based in Ruby, but uses a customized Domain Scripting Language (DSL)
closer to JSON for working within it.
• It runs as a master-client setup and uses a model-driven approach.
• The Puppet code design works as a list of dependencies, which can make things easier or more confusing, depending on your setup.
• When to use it• Puppet is a good choice if stability and maturity are key factors for you. It’s good for large
enterprises with a heterogeneous environment and range of skills on the DevOps team.
• Pricing (subject to change)• Puppet comes in a free open source version
• Paid commercial enterprise version that runs $112 per node per year, with volume discounts.
Source: TaKipi BlogCopyright © William El Kaim 2016 45
Devops Tools Landscape: Puppet
• Pros• Well-established support community through Puppet Labs.
• It has the most mature interface and runs on nearly every OS.
• Simple installation and initial setup.
• Most complete Web UI in this space.
• Strong reporting capabilities.
• Cons• For more advanced tasks, you will need to use the CLI, which is Ruby-based (meaning
you’ll have to understand Ruby).
• Support for pure-Ruby versions (rather than those using Puppet’s customized DSL) is being scaled back.
• Because of the DSL and a design that does not focus on simplicity, the Puppet code base can grow large, unwieldy, and hard to pick up for new people in your organization at higher scale.
• Model-driven approach means less control compared to code-driven approaches.
Source: TaKipi BlogCopyright © William El Kaim 2016 46
Devops Tools Landscape: Saltstack
• SaltStack (or Salt) is a CLI-based tool that can be set up as a master-client model or a non-centralized model. • Based in Python, Salt offers a push method and an SSH method of communication with
clients.
• Salt allows for grouping of clients and configuration templates to simplify the control of the environment.
• When to use it• Salt is a good choice if scalability and resiliency are a big concern. It’s good for system
administrators thanks to its usability.
• Pricing (subject to change)• Free open source version
• SaltStack Enterprise version that is based on an annual per node subscription basis. Specific pricing is not listed on their site (just a “Contact us” link), but others have reported a $150 per node per year starting point.
Source: TaKipi BlogCopyright © William El Kaim 2016 47
Devops Tools Landscape: Saltstack
• Pros
• Straightforward organization and usage once you’re past the setup phase.
• Their DSL is feature-rich and isn’t required for logic and states.
• Input, output, and configs are very consistent – all YAML.
• Introspection is very good. It’s easy to see what’s happening within Salt.
• Strong community.
• High scalability and resiliency in the master model with minions and hierarchical tiers.
• Cons
• Difficult to set up and to pick up for new users.
• Documentation is challenging to understand at the introductory level.
• Web UI is newer and less complete than other tool’s Web UIs in the space.
• Not great support for non-Linux OSs.
Source: TaKipi BlogCopyright © William El Kaim 2016 48
Devops Tools LandscapeAnsible vs. Chef vs. Fabric vs. Puppet vs. SaltStack
Copyright © William El Kaim 2016 49Source: IBM
Devops Tools LandscapeAnsible vs. Chef vs. Fabric vs. Puppet vs. SaltStack
• Which configuration management or deployment automation tool you use will
depend on your needs and preferences for your environment.
• Chef and Puppet are some of the older, more established options, making them good for
larger enterprises and environments that value maturity and stability over simplicity.
• Ansible and SaltStack are good options for those looking for fast and simple solutions
while working in environments that don’t need support for quirky features or lots of OSs.
• Fabric is a good tool for smaller environments and those looking for a more low lift and
entry level solution.
Source: TaKipi BlogCopyright © William El Kaim 2016 50
Devops Tools Landscape: AWS CloudFormation
• AWS CloudFormation gives developers and systems administrators an easy
way to create and manage a collection of related AWS resources.
Copyright © William El Kaim 2016 51
Devops Tools Landscape: Terraform
• Infrastructure as Code
• Infrastructure is described using a high-level
configuration syntax. This allows a blueprint
of your datacenter to be versioned and
treated as you would any other code.
Additionally, infrastructure can be shared and
re-used.
• Execution Plans
• Terraform has a "planning" step where it
generates an execution plan.
• Resource Graph
• Terraform builds a graph of all your
resources, and parallelizes the creation and
modification of any non-dependent resources.
• Change Automation
• Complex change sets can be applied to your
infrastructure with minimal human interaction.
Copyright © William El Kaim 2016 52
Devops Real Examples
• Ansible
• Bower
• Capistrano
• Chef
• Cloud66
• Puppet
• Salstack
• Travis CI / Jenkins CI / Snap-CI
• Vagrant
http://stackshare.io/devopsCopyright © William El Kaim 2016 53
Devops Real Examples
Source: MicrosoftCopyright © William El Kaim 2016 54
Plan
• What is Agile Development?
• Agile Development Requires Continuous Delivery
• What is Devops?
• Devops Tools Landscape
From Devops to Programmatic Infrastructure
Copyright © William El Kaim 2016 55
Rise of Devops
• The rise of DevOps has been part of a three-pronged revolution that is
occurring across industry as businesses grapple with managing new digital
environments:
• An infrastructure revolution is moving much of enterprise to cloud or hybrid cloud
server environments
• An architectural revolution has given rise to APIs, containers and microservices, and
the use of backend-as-a-service products to enable faster creation of single purpose
apps with richer functionality
• An operational revolution has then evolved where these technological revolutions are
changing the way business is organized: business and tech teams are working more
closely together, and every business unit is taking on its IT responsibilities and decision-
making.
56
On The Road To Software-defined Everything
• Amid the fervor surrounding digital, analytics, and cloud, it is easy to
overlook advances currently being made in infrastructure and operations.
• The entire operating environment (server, storage, and network) can now be
virtualized and automated.
• The data center of the future represents the potential for not only lowering
costs, but also dramatically improving speeds and reducing the complexity of
provisioning, deploying, and maintaining technology footprints.
• Software-defined everything can elevate infrastructure investments, from
costly plumbing to competitive differentiators.
Source: DeloitteCopyright © William El Kaim 2016 57
Cloud Taxonomy
Source: BSA Copyright © William El Kaim 2016 58
Cloud Taxonomy
Source: BSA Copyright © William El Kaim 2016 59
CIMI: Cloud IAAS Standard
• Cloud Infrastructure Management Interface
• Specification that standardizes interactions between cloud environments to achieve
interoperable cloud infrastructure management between service providers and their
consumers and developers, enabling users to manage their cloud infrastructure use
easily and without complexity.
• Primer
• Cloud Infrastructure Management Interface Model and RESTful HTTP-based Protocol
Copyright © William El Kaim 2016 60
Cloud Brokers
GravitantCloudOrbitCopyright © William El Kaim 2016 61
Programmatic Infrastructure
• Developers could now program without taking care of the infrastructure and
the different platforms to deploy on
• AWS Elastic Beanstalk lets you deploy your code in seconds.
• AWS CodeDeploy is a continuous delivery and deployment service
• Even the Developer environment (IDE) is now in the Cloud
• Codenvy simplifies setting up environment and running apps
• And the devops could now program the infrastructure like code
• AWS CloudFormation is like « magic » and lets you manage infrastructure with text files
you could store in configuration management tool
• Devops could also reuse Cloud infrastructure and software bricks and assemble them
like legos (like Amazon AWS) or duplos (like BitNami) depending on their granularity.
• Non functional requirements are now available as services
• Everybody could benefit from a world class infrastructure since day one
Copyright © William El Kaim 2016 62
Cloud Cost Advisor
• AWS Simple Monthly Calculator & AWS TCO calculator & AWS Trusted Advisor
• CloudCheckr
• Cloudability
• Cloud Cruiser
Copyright © William El Kaim 2016 63
Codenvy & GitHub: The New Dev Platform
https://codenvy.com/
https://enterprise.github.com/aws
Copyright © William El Kaim 2016 64
Infrastructure As Lego: Amazon AWS
Source: Amazon AWSCopyright © William El Kaim 2016 65
Infrastructure as Lego
Source: Amazon AWSCopyright © William El Kaim 2016 67
Viadeo: No More Datacenter!
Github
CircleCI
Jenkins
AWS S3
AWS AMIspacker.io
cloudformation
Puppet codebase, CloudFormation templates
Copyright © William El Kaim 2016 68
Infrastructure As Lego: Google Cloud Platform
http://googlecloudplatform.blogspot.fr/2015/03/deploy-popular-software-packages-using-Cloud-Launcher.html
https://cloud.google.com/actual-cloud/
Copyright © William El Kaim 2016 69
Infrastructure As Lego: Microsoft Azure
http://azure.microsoft.com/en-us/Copyright © William El Kaim 2016 70
Copyright © William El Kaim 2016 71
Infrastructure As Lego: IBM BlueMix
https://console.ng.bluemix.net/http://www-01.ibm.com/software/bluemix/Copyright © William El Kaim 2016 72
http://www.twitter.com/welkaim
SlideShare
http://www.slideshare.net/welkaim
EA Digital Codex
http://www.eacodex.com/
http://fr.linkedin.com/in/williamelkaim
Claudine O'SullivanCopyright © William El Kaim 2016 73