Embed Size (px)
Citation preview
Automated server managementwith open source tools
Who we are?
● Gawker Media: World’s largest independent media company● We got about ~60M monthly uniques in US, ~100M uniques worldwide.● Our platform called Kinja (http://kinja.com)
Why do we need automation
The work, the great work is now completed,Machine is running, the creator rests.It will move for some millions of yearsTill one of its cogs will need some repairs.
(Imre Madach - The Tragedy of Man)
Automation is good
- Everybody needs automation- DevOps (which is mainly a kind of philosophy, not a thing)- Way less “ops people” than “dev people” at company- People often know what to do, but don’t know how to do it.- The man-hours are expensive, don’t waste them on routine tasks - If a task is automated, there’s less possibility to make a mistake
Familiar?
(This is what doesn’t happen with us!)
Everybody gets involved
● Our developers are responsible for the full life-cycle of their code, including the live environment
● We have to provide them the tools to monitor and control the behavior of the production environment
● We have to do all that without requiring them to ssh connect to the live servers
● We want the devs to be able to do the routine ops tasks
The stack we use
Key elements
- We are running on linux (of course)- Currently we have physical machines (but we consider migrating to the cloud)- We use cobbler for managing the hardware- Puppet for provisioning the hosts- Ansible for running commands on hosts- Jenkins for continous integration
Linux hosts
- De facto standard for internet based companies- Open source- Free
Physical hardware
- We are running on multiple datacenters- All our machines are nearly the same- We can use the same scripts for all the machines
Cobbler
Cobbler is a linux installation server, it covers network install (kickstarting) DNS, DHCP configuration
● All of the hosts are defined in cobbler● No manual DHCP / DNS management● The tool not just manages but also defines the environment
Puppet is a configuration management system that allows you to define the state of your IT infrastructure, then automatically enforces the correct state.
- Manages large set of hosts- Puppet code hosted on github- Defines all the hosts - if it is not in puppet: it doesn’t exists.
Puppet
Ansible
Ansible, a free-software platform for configuring and managing computers, combines multi-node software deployment, ad hoc task execution, and configuration management. It manages nodes over SSH or over PowerShell.
- Just for automation- Commands organised into playbooks- Agentless, easy to install and manage- Open source, you can create your modules, and contribute them to the
community
Architecture
Application server
Replication map
Ansible playbooks
Ansible playbooks
Ansible playbooks
Ansible playbooks
Ansible playbooks
Ansible playbooks - wrapper
Ansible playbooks - wrapper
Jenkins
- Jenkins is a continuous integration and continuous delivery application
- We already used it for building applications- Every project has it’s own workplace, and logs- You can see the history of previous builds- You can share the output of current builds
Jenkins ops jobs
Jenkins job page
Jenkins command output
Jenkins job configuration
Jenkins API
● Trigger jobs with REST API (curl, wget, etc)● Trigger jobs via CLI● Trigger jobs via the state of other jobs
Jenkins-job-builder
● Generates jenkins config from YAML files● You can keep all of your job definitions on GitHub● The Jenkins instance is disposable (we run it as a docker image)
Jenkins-job-builder
The big picture
● We have ansible playbooks which are doing the operations● We have CLI wrappers for all of these playbooks● We turn these wrappers to Jenkins jobs● We run all the commands through Jenkins ● The commands reporting to Slack
