1

ManagingManagingPostgreSQL withPostgreSQL with

AnsibleAnsibleGülçin YıldırımGülçin Yıldırım

FOSDEM PGDay 2016, Brussels

2

select * from me;select * from me;Postgres DBA @

Studying MSc Comp. & Systems Eng. @

Studied BSc Maths Eng. @

Writes blog on 2ndQuadrant Does some childish

Loves independent films

2ndQuadrant

Tallinn University of TechnologyYildiz Technical

Universityblog

paintings@Skype: gulcin2ndq

Github:

apatheticmagpie

gulcin

3

What is Ansible?What is Ansible?“ Simple, agentless and powerful open

source IT automation tool

ProvisioningConfiguration ManagementApplication DeploymentContinuous DeliverySecurity & ComplianceOrchestration

4

What are the options?What are the options?

5

What are the options?What are the options?

6

Why Ansible?Why Ansible?

Agent-less architecture (no agent is required, everything is done byusing SSH, ssh for communication)No centralized server, no client-side agentsSSH basedConfiguration as data, not code (YAML files)Batteries included Full conf. management, deployment and orchestrationCustom modules can be written in any programming language.JSON input/output is sufficient to integrate a module intoAnsible.

7

Life before AnsibleLife before AnsibleMultiple ssh, control panels, editing config files manually.

8

Let's install AnsibleLet's install Ansible

Debian or Ubuntu:Debian or Ubuntu:

apt-get install python-dev python-setuptoolseasy_install pippip install ansible boto

Mac OS X:Mac OS X:

sudo easy_install pipsudo pip install ansible boto

9

Code of this talkCode of this talk

This example:

Provision Amazon VPC and EC2 instancesInstall latest PostgreSQL packagesSetup a streaming replication with 1 master and 2 standbysAdd a new standby server

github.com/gulcin/pgconfeu2015

10

Building BlocksBuilding Blocks Typically, our solutions executes tasks for an inventory, utilizing some modules, using or populating some variables, processing some file templates, in a playbook, which can be organized in roles.

11

InventoryInventoryTells Ansible about hosts it should manage

Hostnames, IPs, ports, SSH parametersServer specific variables

2 common ways to provide Ansible an inventory:Static inventory: A flat INI file (e.g., )Dynamic inventory: An application returning aJSON data (e.g.,: for Amazon EC2)

Hosts are grouped. They can belong to multiplegroups Groups can also belong to multiple groups

hosts.ini

ec2.py

12

InventoryInventoryBelow inventory file in INI format contains 3 hostsunder 2 groups.There is also a 3rd group which contains other groupsand all of the hosts.# "master" group with 1 host[master]postgresql-master ansible_ssh_host=10.0.0.5 ansible_ssh_user=ubuntu

# "standby" group with 2 hosts[standbys]postgresql-standby-01 ansible_ssh_host=10.0.0.10 ansible_ssh_user=ubuntupostgresql-standby-02 ansible_ssh_host=10.0.0.11 ansible_ssh_user=ubuntu

# the "replication" group contains both "master" and "standbys" groups[replication:children]masterstandbys

13

ModuleModuleModules provide Ansible means to control or manageresources on local or remote servers. They perform a variety of functions. For example amodule may be responsible for rebooting a machineor it can simply display a message on the screen. Ansible allows users to write their own modules andalso provides out-of-the-box core or extras modules.

CoreExtras

14

ModuleModuleSome of the most commonly used modules are:

File handling: file, stat, copy, template Remote execution: command, shell Service management: service Package management: apt, yum, bsd, ports Source control systems: git, subversion

15

TaskTaskTasks are responsible for calling a module with a specificset of parameters. Each Ansible task contains:

a descriptive name [optional]a module to be calledmodule parameterspre/post-conditions [optional]processing directives [optional]

They allow us to call Ansible modules and passinformation to consecutive tasks.

16

TaskTaskBelow task invokes the "file" module by providing 4parameters. It ensures 3 conditions are true:

/var/lib/postgresql exists as a directoryowner of /var/lib/postgresql is "postgres"group of /var/lib/postgresql is "postgres"

If it doesn't exist, Ansible creates the directory andassigns owner & group. If only the owner is different,Ansible makes it "postgres".

- name: Ensure the data folder has right ownership file: path="/var/lib/postgresql" state=directory owner=postgres group=postgres

17

TaskTaskFollowing example shows relationships between tasks.The first task checks if a device exists and thesecond task mounts the device depending on the resultfrom the first task.

Please note "register" and "when" keywords.

- name: Check if the data volume partition exists stat: path=/dev/sdc1 register: partition

- name: Ensure the PostgreSQL data volume is mounted mount: src=/dev/sdc1 name="/var/lib/postgresql/9.4" fstype=ext4 state=mounted when: partition.stat.exists is defined and partition.stat.exists

18

VariableVariableVariables in Ansible are very useful for reusinginformation. Sources for variables are:

Inventory: We can assign variables to hosts orgroups (group vars, host vars).YAML files: We can include files containing variables.Task results: Result of a task can be assigned to avariable using the register keyword as shown in theprevious slide.Playbooks: We can define variables in Ansibleplaybooks (more on that later).Command line: (-e means extra variable // -e "uservar=gulcin")

19

VariableVariableThere is also discovered variables (facts) and they can befound by using module:

All of the output in json: bios_version, architecture,default_ipv4_address, ansible_os_family, etc.

You can use variables in tasks, templates themselves andyou can iterate over using with_type functions.

setupansible -i hosts.ini -m setup hostname

- name: Ensure PostgreSQL users are present postgresql_user: state: present name: "{{ item.name }}" password: "{{ item.password }}" role_attr_flags: "{{ item.roles }}" with_items: postgresql_users

20

TemplateTemplateWe can think templates as our configuration files. Ansiblelets us use the for reforming andparameterising our files. The Jinja2 templating engine offers a wide range ofcontrol structures, functions and ..Some of useful capabilities of Jinja2:

for-loopsjoin(), default(), range(), format()union(), intersect(), difference()| to_json, | to_nice_yaml, | from_json, | from_yml| min, | max, | unique, | version_compare, | random

Jinja2 template engine

filters

21

TemplateTemplate

*:*:*:{{ postgresql_replication_user.name }}:{{ postgresql_replication_user.password }}

Let's check our template:pgpass.j2

Let's have a look at file, too:pg_hba.conf.j2

# Access to user from local without password, from subnet with password{% for user in postgresql_users %}local all {{ user.name }} trusthost all {{ user.name }} {{ ec2_subnet }} md5{% endfor %}

# Replication user for standby servers access{% for standby_ip in postgresql_standby_ips %}host replication {{ postgresql_replication_user.name }} {{ standby_ip }}/32 md5{% endfor %}

22

PlaybookPlaybook

Playbooks contains PlaysPlays contain Tasks

Tasks call Modules and may (optionally) triggerhandlers (run once, run at the end)

23

PlaybookPlaybookIf Ansible modules are the tools in your workshop,playbooks are your design plans.

Ansible playbooks are written using the YAML syntax.Playbooks may contain more than one playsEach play contains:

name of host groups to connect totasks it needs to perform.

Strict dependency ordering: everything in fileperforms in a sequential order. (Before v.2)

24

PlaybookPlaybookLet's look at our playbook example ( ):main.yml---- name: Ensure all virtual machines are ready hosts: 127.0.0.1 connection: local vars_files: # load default variables from YAML files below - 'defaults/postgresql.yml' - 'defaults/aws.yml' tasks: - include: 'tasks/provision.yml' # load infrastructure setup tasks

- name: Ensure all required PostgreSQL dependencies ready hosts: postgresql-all # manage all PostgreSQL servers sudo: yes sudo_user: root vars_files: - 'defaults/postgresql.yml' - 'defaults/aws.yml' tasks: - include: 'tasks/postgresql.yml' # load PostgreSQL setup tasks

...

25

RoleRole

“ You absolutely should be using roles. Roles are great.Use roles. Roles! Did we say that enough? Roles are great. In Ansible,

playbooks organize tasksroles organize playbooks

Imagine that we have lots of independent resources tomanage (e.g., web servers, PostgreSQL servers, logging,monitoring, AWS, ...). Putting everything in a single playbook may result in anunmaintainable solution.

26

RoleRoleHere you can see a dependency graph and thecorresponding role directory structure:

27

How to Invoke Ansible?How to Invoke Ansible?

To work with Ansible, we have 2 main alternatives;

1. Running ad-hoc commands2. Running playbooks

Let's check them out one by one.

28

Ad-hoc CommandsAd-hoc CommandsWe can call any Ansible module from the command line, anytime.The ansible CLI tool works like a single task. It requires an inventory,a module name, and module parameters. For example, given an inventory file like:

Now we can call any module.

[dbservers] db.example.com

29

Ad-hoc CommandsAd-hoc CommandsWe can check uptimes of all hosts in dbservers using:

Here we can see the Ansible output:

ansible dbservers -i hosts.ini -m command -a "uptime"

gulcin@apathetic ~ # ansible dbservers -i hosts.ini -m command -a "uptime"db.example.com | success | rc=0 >>21:16:24 up 93 days, 9:17, 4 users, load average: 0.08, 0.03, 0.05

30

How to Run Playbooks?How to Run Playbooks?For more complex scenarios, we can create playbooks or roles andask Ansible to run them. When we run a playbook or a role, Ansible first gathers a lot ofuseful facts about remote systems it manages. These facts can laterbe used in playbooks, templates, config files, etc. We can use the ansible-playbook CLI tool to run playbooks.

31

How to Run Playbooks?How to Run Playbooks?Given an inventory file like this:

We may have a playbook that connects to hosts in dbservers group,executes the uptime command, and then displays that command'soutput. Now let's create a simple playbook to see how it can be ran.

[dbservers] db.example.com

32

How to Run Playbooks?How to Run Playbooks?Here is the main.yml file for the playbook we just described:

---

- hosts: dbservers

tasks: - name: retrieve the uptime command: uptime register: command_result # Store this command's result in this variable

- name: Display the uptime debug: msg="{{ command_result.stdout }}" # Display command output here

33

How to Run Playbooks?How to Run Playbooks?Now we can run the playbook and see it's output here:

gulcin@apathetic ~ $ ansible-playbook -i hosts.ini main.yml

PLAY [dbservers] **************************************************************

GATHERING FACTS ***************************************************************ok: [db.example.com]

TASK: [retrieve the uptime] ***************************************************changed: [db.example.com]

TASK: [Display the uptime] ****************************************************ok: [db.example.com] => { "msg": " 15:54:47 up 3 days, 14:32, 2 users, load average: 0.00, 0.01, 0.05"}

PLAY RECAP ********************************************************************db.example.com : ok=3 changed=1 unreachable=0 failed=0

34

Postgres modulesPostgres modulesMy blog post covers Ansible PostgreSQL modules: You can find the related Postgres modules' examples on my github .

Ansible Loves PostgreSQL

repo

: Creates/removes a given db.

: Adds/removes extensions from a db.

: Adds/removes users and roles from a db.

: Grants/revokes privileges on db objects (table,sequence, function, db, schema, language, tablespace, group).

: Adds, removes or changes procedural languageswith a db.

postgresql_db

postgresql_ext

postgresql_user

postgresql_privs

postgresql_lang

35

postgresql_dbpostgresql_dbCreates/removes a given db. In Ansible terminology, it ensures thata given db is present or absent.

Required param: nameConnection params: login_host, port,login_user, login_passwordImportant param: state (present or absent)

Create db module_test Remove db module_test

36

postgresql_extpostgresql_extAdds/removes extensions from a db. PostgreSQL offers a wide rangeof extensions which are extremely helpful.

Mandatory params: db and name (for extension)Other params: state and connection params (login_host, port) asin postgres_db

37

postgresql_userpostgresql_userAdds/removes users and roles from a db. Parameters: name (mandatory), state, connection params, db,password, priv, role_attr_flags ([NO]SUPERUSER, [NO]CREATEDB..)

Creates userAlters role and giveslogin and createdb

38

postgresql_privspostgresql_privsGrants/revokes privileges on db objects (table, sequence, function, db,schema, language, tablespace, group).

Required params: database, rolesImportant opt. params: type (i.e. table,sequence,function), objs,privs (i.e. ALL; SELECT, UPDATE, INSERT)

39

postgresql_langpostgresql_lang Adds, removes or changes procedural languages with a db.

“ One of the very powerful features of PostgreSQL is its support forvirtually any language to be used as a procedural language.

Params: lang (mandatory), db, connection params, state,cascade, trust

40

AWS modulesAWS modulesAnsible provides more than 50 modules for provisioningand managing Amazon Web Services resources.

We have modules for:

EC2 (Virtual machine instances)AMI (Virtual machine images)ELB (Load balancers)VPC (Networking infrastructure)EBS (Storage infrastructure)...

http://docs.ansible.com/ansible/list_of_cloud_modules.html

41

AWS modulesAWS modulesAnsible AWS modules are developed using .Boto is a Python package that provides interfaces to Amazon WebServices. For more complex workflows, we can use the script todiscover dynamic inventories for our playbooks:

Download ec2.py and ec2.inifrom cp ec2.py /etc/ansible/hostscp ec2.ini /etc/ansiblechmod +x /etc/ansible/hosts

Boto

ec2.py

github.com/ansible/ansible/tree/devel/contrib/inventory

42

Playbook structurePlaybook structure

43

Playbook AWS architecturePlaybook AWS architecture

44

Action! Action!

45

First runFirst run

46

Verify setupVerify setup

47

New standby serverNew standby server

48

ConclusionConclusion

Ansible loves PostgreSQL and Ansible has a very active community.<wishlist>

That's why more PostgreSQL modules would be helpful for everyone.

Making contributions to Ansible will be appreciated :)</wishlist>

49

Questions? Questions?

Huge Thanks!Huge Thanks!

50

ReferencesReferencesAnsible quick start video

Review: Puppet vs Chef vs Ansible vs Salt

Managing PostgreSQL with Ansible in EC2

Jinja2 for better Ansible playbooks and templates

Edmund The Elephant

http://www.ansible.com/videos

http://www.infoworld.com/article/2609482/data-

center/data-center-review-puppet-vs-chef-vs-ansible-vs-salt.html

https://www.safaribooksonline.com/library/view/velocity-conference-

2013/9781449371630/part22.html

https://blog.codecentric.de/en/2014/08/jinja2-better-ansible-playbooks-templates/

https://dribbble.com/shots/2223604-Edmund-The-Elephant