1
ManagingManagingPostgreSQL withPostgreSQL with
AnsibleAnsibleGülçin YıldırımGülçin Yıldırım
FOSDEM PGDay 2016, Brussels
2
select * from me;select * from me;Postgres DBA @
Studying MSc Comp. & Systems Eng. @
Studied BSc Maths Eng. @
Writes blog on 2ndQuadrant Does some childish
Loves independent films
2ndQuadrant
Tallinn University of TechnologyYildiz Technical
Universityblog
paintings@Skype: gulcin2ndq
Github:
apatheticmagpie
gulcin
3
What is Ansible?What is Ansible?“ Simple, agentless and powerful open
source IT automation tool
ProvisioningConfiguration ManagementApplication DeploymentContinuous DeliverySecurity & ComplianceOrchestration
4
What are the options?What are the options?
5
What are the options?What are the options?
6
Why Ansible?Why Ansible?
Agent-less architecture (no agent is required, everything is done byusing SSH, ssh for communication)No centralized server, no client-side agentsSSH basedConfiguration as data, not code (YAML files)Batteries included Full conf. management, deployment and orchestrationCustom modules can be written in any programming language.JSON input/output is sufficient to integrate a module intoAnsible.
7
Life before AnsibleLife before AnsibleMultiple ssh, control panels, editing config files manually.
8
Let's install AnsibleLet's install Ansible
Debian or Ubuntu:Debian or Ubuntu:
apt-get install python-dev python-setuptoolseasy_install pippip install ansible boto
Mac OS X:Mac OS X:
sudo easy_install pipsudo pip install ansible boto
9
Code of this talkCode of this talk
This example:
Provision Amazon VPC and EC2 instancesInstall latest PostgreSQL packagesSetup a streaming replication with 1 master and 2 standbysAdd a new standby server
github.com/gulcin/pgconfeu2015
10
Building BlocksBuilding Blocks Typically, our solutions executes tasks for an inventory, utilizing some modules, using or populating some variables, processing some file templates, in a playbook, which can be organized in roles.
11
InventoryInventoryTells Ansible about hosts it should manage
Hostnames, IPs, ports, SSH parametersServer specific variables
2 common ways to provide Ansible an inventory:Static inventory: A flat INI file (e.g., )Dynamic inventory: An application returning aJSON data (e.g.,: for Amazon EC2)
Hosts are grouped. They can belong to multiplegroups Groups can also belong to multiple groups
hosts.ini
ec2.py
12
InventoryInventoryBelow inventory file in INI format contains 3 hostsunder 2 groups.There is also a 3rd group which contains other groupsand all of the hosts.# "master" group with 1 host[master]postgresql-master ansible_ssh_host=10.0.0.5 ansible_ssh_user=ubuntu
# "standby" group with 2 hosts[standbys]postgresql-standby-01 ansible_ssh_host=10.0.0.10 ansible_ssh_user=ubuntupostgresql-standby-02 ansible_ssh_host=10.0.0.11 ansible_ssh_user=ubuntu
# the "replication" group contains both "master" and "standbys" groups[replication:children]masterstandbys
13
ModuleModuleModules provide Ansible means to control or manageresources on local or remote servers. They perform a variety of functions. For example amodule may be responsible for rebooting a machineor it can simply display a message on the screen. Ansible allows users to write their own modules andalso provides out-of-the-box core or extras modules.
CoreExtras
14
ModuleModuleSome of the most commonly used modules are:
File handling: file, stat, copy, template Remote execution: command, shell Service management: service Package management: apt, yum, bsd, ports Source control systems: git, subversion
15
TaskTaskTasks are responsible for calling a module with a specificset of parameters. Each Ansible task contains:
a descriptive name [optional]a module to be calledmodule parameterspre/post-conditions [optional]processing directives [optional]
They allow us to call Ansible modules and passinformation to consecutive tasks.
16
TaskTaskBelow task invokes the "file" module by providing 4parameters. It ensures 3 conditions are true:
/var/lib/postgresql exists as a directoryowner of /var/lib/postgresql is "postgres"group of /var/lib/postgresql is "postgres"
If it doesn't exist, Ansible creates the directory andassigns owner & group. If only the owner is different,Ansible makes it "postgres".
- name: Ensure the data folder has right ownership file: path="/var/lib/postgresql" state=directory owner=postgres group=postgres
17
TaskTaskFollowing example shows relationships between tasks.The first task checks if a device exists and thesecond task mounts the device depending on the resultfrom the first task.
Please note "register" and "when" keywords.
- name: Check if the data volume partition exists stat: path=/dev/sdc1 register: partition
- name: Ensure the PostgreSQL data volume is mounted mount: src=/dev/sdc1 name="/var/lib/postgresql/9.4" fstype=ext4 state=mounted when: partition.stat.exists is defined and partition.stat.exists
18
VariableVariableVariables in Ansible are very useful for reusinginformation. Sources for variables are:
Inventory: We can assign variables to hosts orgroups (group vars, host vars).YAML files: We can include files containing variables.Task results: Result of a task can be assigned to avariable using the register keyword as shown in theprevious slide.Playbooks: We can define variables in Ansibleplaybooks (more on that later).Command line: (-e means extra variable // -e "uservar=gulcin")
19
VariableVariableThere is also discovered variables (facts) and they can befound by using module:
All of the output in json: bios_version, architecture,default_ipv4_address, ansible_os_family, etc.
You can use variables in tasks, templates themselves andyou can iterate over using with_type functions.
setupansible -i hosts.ini -m setup hostname
- name: Ensure PostgreSQL users are present postgresql_user: state: present name: "{{ item.name }}" password: "{{ item.password }}" role_attr_flags: "{{ item.roles }}" with_items: postgresql_users
20
TemplateTemplateWe can think templates as our configuration files. Ansiblelets us use the for reforming andparameterising our files. The Jinja2 templating engine offers a wide range ofcontrol structures, functions and ..Some of useful capabilities of Jinja2:
for-loopsjoin(), default(), range(), format()union(), intersect(), difference()| to_json, | to_nice_yaml, | from_json, | from_yml| min, | max, | unique, | version_compare, | random
Jinja2 template engine
filters
21
TemplateTemplate
*:*:*:{{ postgresql_replication_user.name }}:{{ postgresql_replication_user.password }}
Let's check our template:pgpass.j2
Let's have a look at file, too:pg_hba.conf.j2
# Access to user from local without password, from subnet with password{% for user in postgresql_users %}local all {{ user.name }} trusthost all {{ user.name }} {{ ec2_subnet }} md5{% endfor %}
# Replication user for standby servers access{% for standby_ip in postgresql_standby_ips %}host replication {{ postgresql_replication_user.name }} {{ standby_ip }}/32 md5{% endfor %}
22
PlaybookPlaybook
Playbooks contains PlaysPlays contain Tasks
Tasks call Modules and may (optionally) triggerhandlers (run once, run at the end)
23
PlaybookPlaybookIf Ansible modules are the tools in your workshop,playbooks are your design plans.
Ansible playbooks are written using the YAML syntax.Playbooks may contain more than one playsEach play contains:
name of host groups to connect totasks it needs to perform.
Strict dependency ordering: everything in fileperforms in a sequential order. (Before v.2)
24
PlaybookPlaybookLet's look at our playbook example ( ):main.yml---- name: Ensure all virtual machines are ready hosts: 127.0.0.1 connection: local vars_files: # load default variables from YAML files below - 'defaults/postgresql.yml' - 'defaults/aws.yml' tasks: - include: 'tasks/provision.yml' # load infrastructure setup tasks
- name: Ensure all required PostgreSQL dependencies ready hosts: postgresql-all # manage all PostgreSQL servers sudo: yes sudo_user: root vars_files: - 'defaults/postgresql.yml' - 'defaults/aws.yml' tasks: - include: 'tasks/postgresql.yml' # load PostgreSQL setup tasks
...
25
RoleRole
“ You absolutely should be using roles. Roles are great.Use roles. Roles! Did we say that enough? Roles are great. In Ansible,
playbooks organize tasksroles organize playbooks
Imagine that we have lots of independent resources tomanage (e.g., web servers, PostgreSQL servers, logging,monitoring, AWS, ...). Putting everything in a single playbook may result in anunmaintainable solution.
26
RoleRoleHere you can see a dependency graph and thecorresponding role directory structure:
27
How to Invoke Ansible?How to Invoke Ansible?
To work with Ansible, we have 2 main alternatives;
1. Running ad-hoc commands2. Running playbooks
Let's check them out one by one.
28
Ad-hoc CommandsAd-hoc CommandsWe can call any Ansible module from the command line, anytime.The ansible CLI tool works like a single task. It requires an inventory,a module name, and module parameters. For example, given an inventory file like:
Now we can call any module.
[dbservers] db.example.com
29
Ad-hoc CommandsAd-hoc CommandsWe can check uptimes of all hosts in dbservers using:
Here we can see the Ansible output:
ansible dbservers -i hosts.ini -m command -a "uptime"
gulcin@apathetic ~ # ansible dbservers -i hosts.ini -m command -a "uptime"db.example.com | success | rc=0 >>21:16:24 up 93 days, 9:17, 4 users, load average: 0.08, 0.03, 0.05
30
How to Run Playbooks?How to Run Playbooks?For more complex scenarios, we can create playbooks or roles andask Ansible to run them. When we run a playbook or a role, Ansible first gathers a lot ofuseful facts about remote systems it manages. These facts can laterbe used in playbooks, templates, config files, etc. We can use the ansible-playbook CLI tool to run playbooks.
31
How to Run Playbooks?How to Run Playbooks?Given an inventory file like this:
We may have a playbook that connects to hosts in dbservers group,executes the uptime command, and then displays that command'soutput. Now let's create a simple playbook to see how it can be ran.
[dbservers] db.example.com
32
How to Run Playbooks?How to Run Playbooks?Here is the main.yml file for the playbook we just described:
---
- hosts: dbservers
tasks: - name: retrieve the uptime command: uptime register: command_result # Store this command's result in this variable
- name: Display the uptime debug: msg="{{ command_result.stdout }}" # Display command output here
33
How to Run Playbooks?How to Run Playbooks?Now we can run the playbook and see it's output here:
gulcin@apathetic ~ $ ansible-playbook -i hosts.ini main.yml
PLAY [dbservers] **************************************************************
GATHERING FACTS ***************************************************************ok: [db.example.com]
TASK: [retrieve the uptime] ***************************************************changed: [db.example.com]
TASK: [Display the uptime] ****************************************************ok: [db.example.com] => { "msg": " 15:54:47 up 3 days, 14:32, 2 users, load average: 0.00, 0.01, 0.05"}
PLAY RECAP ********************************************************************db.example.com : ok=3 changed=1 unreachable=0 failed=0
34
Postgres modulesPostgres modulesMy blog post covers Ansible PostgreSQL modules: You can find the related Postgres modules' examples on my github .
Ansible Loves PostgreSQL
repo
: Creates/removes a given db.
: Adds/removes extensions from a db.
: Adds/removes users and roles from a db.
: Grants/revokes privileges on db objects (table,sequence, function, db, schema, language, tablespace, group).
: Adds, removes or changes procedural languageswith a db.
postgresql_db
postgresql_ext
postgresql_user
postgresql_privs
postgresql_lang
35
postgresql_dbpostgresql_dbCreates/removes a given db. In Ansible terminology, it ensures thata given db is present or absent.
Required param: nameConnection params: login_host, port,login_user, login_passwordImportant param: state (present or absent)
Create db module_test Remove db module_test
36
postgresql_extpostgresql_extAdds/removes extensions from a db. PostgreSQL offers a wide rangeof extensions which are extremely helpful.
Mandatory params: db and name (for extension)Other params: state and connection params (login_host, port) asin postgres_db
37
postgresql_userpostgresql_userAdds/removes users and roles from a db. Parameters: name (mandatory), state, connection params, db,password, priv, role_attr_flags ([NO]SUPERUSER, [NO]CREATEDB..)
Creates userAlters role and giveslogin and createdb
38
postgresql_privspostgresql_privsGrants/revokes privileges on db objects (table, sequence, function, db,schema, language, tablespace, group).
Required params: database, rolesImportant opt. params: type (i.e. table,sequence,function), objs,privs (i.e. ALL; SELECT, UPDATE, INSERT)
39
postgresql_langpostgresql_lang Adds, removes or changes procedural languages with a db.
“ One of the very powerful features of PostgreSQL is its support forvirtually any language to be used as a procedural language.
Params: lang (mandatory), db, connection params, state,cascade, trust
40
AWS modulesAWS modulesAnsible provides more than 50 modules for provisioningand managing Amazon Web Services resources.
We have modules for:
EC2 (Virtual machine instances)AMI (Virtual machine images)ELB (Load balancers)VPC (Networking infrastructure)EBS (Storage infrastructure)...
http://docs.ansible.com/ansible/list_of_cloud_modules.html
41
AWS modulesAWS modulesAnsible AWS modules are developed using .Boto is a Python package that provides interfaces to Amazon WebServices. For more complex workflows, we can use the script todiscover dynamic inventories for our playbooks:
Download ec2.py and ec2.inifrom cp ec2.py /etc/ansible/hostscp ec2.ini /etc/ansiblechmod +x /etc/ansible/hosts
Boto
ec2.py
github.com/ansible/ansible/tree/devel/contrib/inventory
42
Playbook structurePlaybook structure
43
Playbook AWS architecturePlaybook AWS architecture
44
Action! Action!
45
First runFirst run
46
Verify setupVerify setup
47
New standby serverNew standby server
48
ConclusionConclusion
Ansible loves PostgreSQL and Ansible has a very active community.<wishlist>
That's why more PostgreSQL modules would be helpful for everyone.
Making contributions to Ansible will be appreciated :)</wishlist>
49
Questions? Questions?
Huge Thanks!Huge Thanks!
50
ReferencesReferencesAnsible quick start video
Review: Puppet vs Chef vs Ansible vs Salt
Managing PostgreSQL with Ansible in EC2
Jinja2 for better Ansible playbooks and templates
Edmund The Elephant
http://www.ansible.com/videos
http://www.infoworld.com/article/2609482/data-
center/data-center-review-puppet-vs-chef-vs-ansible-vs-salt.html
https://www.safaribooksonline.com/library/view/velocity-conference-
2013/9781449371630/part22.html
https://blog.codecentric.de/en/2014/08/jinja2-better-ansible-playbooks-templates/
https://dribbble.com/shots/2223604-Edmund-The-Elephant