With Ansible and Git Umut Bozkurt, 20.11...Standardizing Oracle Environment With Ansible and Git Umut Bozkurt, 20.11.2019

Standardizing Oracle Environment

With Ansible and Git

Umut Bozkurt, 20.11.2019

−Problem

− Increasing number of databases

− Increasing security requirements

− Consistent number of DBAs

−Solution

− Increasing number of DBAs

− Standardization

− Automation

30

40

50

60

70

80

90Oracle Database Size (TB)

About me…

17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank4

−Umut Bozkurt

− Database Engineer

− Swiss National Bank

−More than 18 years of Oracle Database experience

− from database development to Oracle hardware

Integrating Ansible Tower

Oracle Infrastructure

−Current platform: IBM AIX, LPAR

−Next platform: Red Hat, VMware ESX

−Oracle technologies

− Single instance

− Multitenant

− Data Guard

− Automatic Storage Management

− Oracle Enterprise Manager

− Automation with shell scripts


What is Ansible?

− Ansible is an automation language and engine

− Ansible is agentless and uses OpenSSH or WinRM

− Ansible terminology:

− Playbooks files contains plays, in YAML

− YAML is a data serialization standard that can be used in

conjunction with all programming languages

− Plays consists of tasks and ensures that managed hosts

are in a particular state

− Tasks calls modules

− Tasks runs sequentially

− Playbooks are executed from the control node

− Managed hosts are listed in the inventory


- name: httpd server installation

hosts: all

become: yes

become_user: root

tasks:

- name: httpd server is installed

yum:

name: httpd

state: present

- name: httpd is started and enabled

service:

name: httpd

state: started

enabled: true

What is Ansible Tower?

− Ansible Tower is a web console and REST API

− Ansible Tower is a commercial offering from Red Hat

− Ansible AWX is the upstream project of the Ansible Tower

− Ansible Tower features:

− Role based access control

− Job scheduling

− Workflows

− Credential management

− Logging and auditing

− Real time and historical job status reporting

− Notifications

− REST API


Ansible Tower Organization

−Credentials

− Git repository credentials

− Server credentials

− Projects

− Git connection

− Git credential

− Git branch

− Inventories

− Managed hosts

− Groups

− Variables

Templates


Security Concept - Personal Users


Authentication

AD account 1 +

Smart Card certificate

Authentication

AD account 2 +

Smart Card certificate

Authentication

Unix account + RSA SecurID

Authorization

Database Server privileges

Change Management

># Database Server># ---------------> logged in as umut> sudo su - oracle

># Database Server># ---------------> logged in as umut> sudo su -

># Database Server># ---------------> logged in as umut> sudo su -

># Jump Host># ---------------

> ssh umut@dbsrv

Authentication

AD account 3

−Running Ad-Hoc commands only with the personal accounts

Security Concept - Jobs

− Scheduling jobs only with a technical user and a privileged

credential

− Scheduler user

− Only execute privileges on the assigned template

−Credential

− Encrypted and stored in the Ansible Tower database

− Can only be used from the Tower GUI or API

− Exclusive for the scheduler user


# Database Server# ---------------> cat authorized_users...from "10.24.23.11" ssh-rsa AKCB3NzaC1yc...

# Database Server# ---------------> cat sshd_config... AllowUsers [email protected]...

# Database Server# ---------------> sudo su -

Security Concept - Operations


− Execute privileges on a template allows running the template. Viewing or modifications are

not possible

− Surveys allows operations team or end users to run the templates with input parameters

Standardizing Oracle Environment

Standardization

− IT standardization is a strategy for minimizing the IT

costs by keeping the hardware and software as

consistent as possible

−Our goal is to consistently push and (if changed)

overwrite both our scripts and configuration files to the

database servers

− This will lead us to

− easy automation

− increase security

− reduce human errors


># Development Server># ------------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

Standardization with Ansible and Git

−Git

− Playbooks, inventory scripts

− Files to push to the servers

− LDAP

− Server and lifecycle information

−CMDB

− Oracle instance information

−Oracle Enterprise Manager

− Monitoring


># Database Server># ----------------

># Database Server># ----------------

># Database Server># ----------------

LDAP CMDB

># Ansible Tower ># Rest API># ---------------

curl -X GET …

Git Projects

Files to push to the servers


oracle_distribution

oracle_automation

Scripts for automation, like playbooks,

dynamic inventory scripts,..

Protected branches cannot be deleted and cannot be force pushed

Inventory

−Managed hosts are defined in the inventory

−Hosts are organized by groups

− The inventory can be defined either

− in a static text file

− dynamically by scripts, from the external sources


# My Static Inventory# -------------------

[PROD]

dbserver01

dbserver02

dbserver03

[TEST]

dbserver05

dbserver06

[AVALOQ]

dbserver01

dbserver05

[SAP]

dbserver03

dbserver06

Instance Information in the Inventory

− In most cases, extending the inventory with the instance

information is helpful

− Instance information can be gathered either

− directly from the managed hosts (custom facts)

− from the external inventories


# My Static Inventory# -------------------

[PROD]

dbserver01 instance=AAAL HOME=/u01/...

dbserver02 instance=DB01 HOME=/u01/...

dbserver03 instance=PRQ HOME=/PRQ/...

[TEST]

dbserver05 instance=AAAQ HOME=/u01/...

dbserver06 instance=SRQ HOME=/SRQ/...

[AVALOQ]

dbserver01 instance=AAAL HOME=/u01/...

dbserver05 instance=AAAQ HOME=/u01/...

[SAP]

dbserver03 instance=PRQ HOME=/PRQ/...

dbserver06 instance=SRQ HOME=/SRQ/...

Ansible Facts

− Facts are

− variables that are automatically discovered on the

managed host

− collected before executing the first task

− Fact information can be accessed later in the tasks


PLAY [Install Oracle PDB] *************

TASK [Gathering Facts]***************

ok: [birs1z.snb.ch]

TASK [Create PDB] *******************

changed: [birs1z.snb.ch]

PLAY [Install Oracle PDB] *************

TASK [Gathering Facts]***************

ok: [server1.ubozkurt.com]

TASK [Create PDB] *******************

changed: [server1.ubozkurt.com]

Custom Facts


−Custom facts

− allows us to gather additional information (like instance information)

from the managed hosts

− are either static files or scripts stored in the managed host

−Custom fact scripts must

− be executable

− have the extension ".fact”

− output in JSON

− be placed in /etc/ansible/facts.d directory on the managed host

>./local_script.fact

{"instances": [

{"name": "DBMS01CD","oracle_home": "…","version": "19c"

},{

"name": "DBMS02CD","oracle_home": "…","version": "18c"

}]

}

Dynamic Inventory


− Inventory scripts

− output in JSON

− mandatory arguments:

− --list

− --host <hostname>

− runs on the Ansible controller

− does not accept input parameters. But

you can work with environment

variables

> ./snb_invent.py --list

{“production": {

"hosts": ["prdserver1.snb.ch","prdserver2.snb.ch"

],"vars": {}

},“development": {

"hosts": ["devserver3.snb.ch"

]"vars": {}

}}

> ./snb_invent.py \--host prdserver2.snb.ch

{"instances": [

{"name": "DBMS01CD","oracle_home": "…“,"version": "19c"

},{

"name": "DBMS02CD","oracle_home": "…“,"version": "18c"

}]

}

Clone Git Repository to the Ansible Tower


># Ansible Tower>------------------

> ls -la tower_fs/oracle_distribution

.gitu01_app_oraclehome_oracle

- name: Clone Git project oracle_distribution

hosts: localhost

connection: local

gather_facts: false

tasks:

- name: Clone dev branch to Ansible Tower file system

git:

repo: 'http://ansitower:vk6z@gitlab/dbms/oracle_distribution.git'

dest: /tower_fs/

version: dev

force: yes

depth: 1


># Database Server>------------------

> ls -l /u01/app/oracle/snb

basenvbindbsetchistoryinstalllog -> /var/log/dbmsrcvsqltmp

- name: Oracle File Distribution

hosts: all

become: yes

become_user: oracle

gather_facts: true

gather_subset: min

tasks:

- name: Synchronize from the Tower file system to the managed host

synchronize:

src: "/tower_fs/oracle_distribution/u01_app_oracle/snb/{{ item }}"

dest: "/u01/app/oracle/snb/"

checksum: yes

delete: yes

times: yes

recursive: yes

rsync_opts: "--chown=oracle:dba"

loop:

- basenv

- bin

- install

- rcv

- sql

Distribute files from Ansible Tower to the Servers

Configuration files with Jinja2 Templates


− A template contains variables which are replaced by their values when the template is

rendered

− Ansible uses Jinja2 templating

<title>{% block title %}{% endblock %}</title><ul>{% for user in users %}

<li><a href="{{ user.url }}">{{ user.username }}</a></li>{% endfor %}</ul>

listener.ora with Jinja2 Templates


# Jinja2 Template # -----------------------------------LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST={{ ansible_fqdn }})(PORT=1599)))

SID_LIST_LISTENER=(SID_LIST={% for instance in hostvars[inventory_hostname].instances %}(SID_DESC=(ORACLE_HOME={{ instance.oracle_home }})(SID_NAME={{ instance.name }})){% endfor %})

# Listener.ora# -----------------------------------------------------------------LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbserver.snb.ch)(PORT=1599)))

SID_LIST_LISTENER=(SID_LIST=(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home19c)(SID_NAME=DBMS01CD))(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home19c)(SID_NAME=DBMS02CD))(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home18c)(SID_NAME=DBMS03CD)))

# Output of the Dynamic Inventory # Script# -------------------------------

> ./snb_invent.py --host dbserver.snb.ch{"instances": [{"name": "DBMS01CD","oracle_home": "…","version": "19c"

},{"name": "DBMS02CD","oracle_home": "…","version": "19c"

},{"name": "DBMS03CD","oracle_home": "…","version": "18c"

}]}

# Task in the playbook# ------------------------------------ name: creating listener.ora

template:src: "/tower_fs/oracle_distribution/db00_app_oracle/snb/templates/listener.j2"dest: "/u01/app/oracle/network/admin/listener.ora"owner: oraclegroup: dbamode: '0640'force: yes

Automation with Ansible

Modules


- name: httpd server installation

hosts: all

become: yes

become_user: root

tasks:

- name: httpd server is installed

yum:

name: httpd

state: present

- name: httpd is started and enabled

service:

name: httpd

state: started

enabled: true

># Ansible Tower

># -------------

> cat /usr/lib/python2.7/site-packages/ansible/modules/system/service.py

#!/usr/bin/python

...

options:

name:

description:

- Name of the service.

type: str

required: true

state:

description:

- C(started)/C(stopped) are idempotent actions that will not run

commands unless necessary.

- C(restarted) will always bounce the service.

- C(reloaded) will always reload.

- B(At least one of state and enabled are required.)

type: str

choices: [ reloaded, restarted, started, stopped ]

...

Module Support

− Four types of modules:

− Core: Maintained and supported by Ansible

engineering team

− Network: Maintained and supported by Ansible

Network team

− Certified: Maintained by Red Hat partners. First

level contact is Red Hat

− Community: Not supported under an Ansible

engine subscription


Documentation of the synchronize module

−Organizations requires regulations about using the community modules

Oracle Modules

−One of the most popular community module for Oracle is

Ansible-Oracle-Modules, developed by Mikael Sandström

− Can be downloaded from GitHub

−Oracle also provides Oracle Cloud Infrastructure Ansible

modules


> ls ansible-oracle-modules

oracle_asmdgoracle_asmvoloracle_awroracle_datapatchoracle_dboracle_factsoracle_gi_factsoracle_grantsoracle_joboracle_jobclassoracle_jobscheduleoracle_jobwindoworacle_ldapuseroracle_opatchoracle_parameteroracle_pdboracle_privsoracle_profileoracle_redooracle_roleoracle_rsrc_consgrouporacle_servicesoracle_sqloracle_stats_prefsoracle_tablespaceoracle_user

− Automating database tasks without logging in to each

server

− Database software installation

− Database creation

− Database configuration

− Database patching and upgrade

− Database cloning

− ...

− Base for a self-service platform


># Database Server># ---------------

> Creating User

># Database Server># ---------------

> Cloning PDB

># Database Server># ---------------

> Creating Database

Self Service Portal


curl -X GET …

Oracle Infrastructure Automation with Ansible

># Database Server># ---------------

> Creating User


curl -X GET …

Applying Release Update


- name: Start the instance and apply datapatch

shell: |

export ORACLE_HOME="{{ item.oracle_home }}"

export ORACLE_SID="{{ item.name }}"

$ORACLE_HOME/bin/sqlplus / as sysdba << EOF

startup

EOF

$ORACLE_HOME/OPatch/datapatch -verbose

register: shell_result

changed_when: >

( "'Installing patches...' in shell_result.stdout" ) or

( "'ORACLE instance started.' in shell_result.stdout" )

failed_when: "'ORA-' in shell_result.stdout"

with_items: "{{ hostvars[inventory_hostname]['instances'] }}"

when: item.oracle_home == "/db00/app/product/db19c/db00"

- name: Copy RU from the control host to the servers and unzip

unarchive:

copy: yes

src: "/ansible_fs/p30125133_190000_Linux-x86-64.zip"

dest: "/db00/app/snb/tmp_for_apply_psu"

owner: oracle

group: dba

- name: Apply RU with Ansible-Oracle-Modules

oracle_opatch:

oracle_home: "/db00/app/oracle/product/db19c/db00"

patch_base: "/db00/app/snb/tmp_for_apply_psu/30125133"

opatch_minversion: '12.2.0.1.17'

conflict_check: yes

stop_processes: no

state: present

Ansible Execution


># Ansible Tower># ---------------

> ansible-playbook db.yml -e "oracle_sid=orcl"

># Managed Host># ----------------

> 1. create directory in ~/.ansible/tmp/…> 2. copy module in the created directory> 3. execute> 4. send results in JSON format> 5. remove temporary directory

SSH

−Most of the modules requires python installed on the managed host. Modules can also

have additional requirements

− All playbooks are executed as awx user on the control node

># Managed Host># ----------------

> 1. create directory in ~/.ansible/tmp/…> 2. copy module in the created directory> 3. execute> 4. send results in JSON format> 5. remove temporary directory

Developing Ansible Modules

− You can use any language; however python is the most suitable

− Ansible modules strive for idempotency:

− You can run a playbook on the same host multiple times. When your systems are in the correct

state, the playbook should not make any changes

− Idempotency is not always possible especially when you are calling external programs, like

oracle setup or grid infrastructure root scripts


> mkdir /u01/app

> If (! -d /u01/app); then mkdir /u01/app ; fi

x

Simple Create/Remove Directory Module with bash and python

−Default custom module directory in Ansible Tower is

/usr/share/ansible/plugins/modules

− Ansible calls custom scripts with a parameter file

−Output should be a JSON object

− “failed” should be true in case of a failure; exit value is

not enough

− “changed” key is for idempotency


> ./my_bash_script.ksh /tmp/variables

{ "changed" : false,"msg": "ERROR:cannot create directory…","failed": true }

- name:create directory with a bash modulemy_bash_script:

dirname: "/tmp/new_dir"state: present

#!/usr/bin/ksh

source $1

echo ${dirname}echo ${state}



#!/usr/bin/bash

source $1

typeset -l state=${state:-present}

if !([ "${state}" = "absent" ] || [ "${state}" = "present" ]) then

echo "{ \"failed\": true,"

echo "\"msg\": \"ERROR: state must be absent or present\"}"

exit 1

fi

if [ "${dirname}" = "" ] ; then

echo "{ \"failed\": true, "

echo " \"msg\": \"ERROR: please provide dirname \" }"

exit 1

fi

#!/usr/bin/python

import os

def main():

module = AnsibleModule(

argument_spec = dict(

dirname = dict(required=True, type='str'),

state=dict(default='present',choices=['present','absent'])

)

)

dirname = module.params["dirname"]

state = module.params["state"]



if [ "${state}" = "absent" ] && [ -d "${dirname}" ] ; then

vRetMsg=$( rm -rf ${dirname} 2>&1 )

if [ $? -eq 0 ] ; then

echo "{ \"changed\": true }"

else


echo " \"msg\": \"ERROR: ${vRetMsg}\" }"

exit 1

fi

elif [ "${state}" = "present" ] && [ ! -d "${dirname}" ] then

vRetMsg=$( mkdir ${dirname} 2>&1 )

if [ $? -eq 0 ] ; then

echo "{ \"changed\": true }"

else


echo " \"msg\": \"ERROR: ${vRetMsg}\" }"

exit 1

fi

else

echo " { \"changed\": false }"

fi

exit 0

if state == 'absend' and os.path.exists(dirname):

try:

os.rmdir(dirname)

except OSError as err:

module.fail_json(msg="OS error: {0}".format(err))

else:

module.exit_json(changed=True)

elif state == 'present' and not os.path.exists(dirname):

try:

os.mkdir(dirname)

except OSError as err:

module.fail_json(msg="OS error: {0}".format(err))

else:

module.exit_json(changed=True)

else:

module.exit_json(changed=False)

from ansible.module_utils.basic import *

if __name__ == '__main__':

main()



TASK [Create directory] ************************************************

changed: [srv1.ubozkurt.com]

TASK [Create directory second time] ************************************************

ok: [srv1.ubozkurt.com]

TASK [Create directory without having permission on the fs] ************************************************

fatal: [srv1.ubozkurt.com]: FAILED! => {"changed": false, "msg": "ERROR: mkdir: cannot create directory ‘/umut_new_dir’: Permission denied"}

...ignoring

TASK [Invalid input parameter] ************************************************

fatal: [srv1.ubozkurt.com]: FAILED! => {"changed": false, "msg": "ERROR: state must be absent or present"}

...ignoring

tasks:

- name: Create directory

my_bash_script:

dirname: "/tmp/umut_new_dir"

state: present

- name: Create directory second time

my_bash_script :


state: present

- name: Create directory without having permission on the fs

my_bash_script:

dirname: "/umut_new_dir"

state: present

ignore_errors: true

- name: Invalid input parameter

my_bash_script:


state: "invalid_value"

ignore_errors: true

Summary

− Ansible is a simple, powerful automation engine

− There are many use cases for automating daily tasks

−Role based access control helps to integrate Ansible Tower in to the organizations

−Organization wide usage requires a good security concept


© Swiss National Bank

Thank you for your attention!

Documents

With Ansible and Git Umut Bozkurt, 20.11...Standardizing Oracle Environment With Ansible and Git Umut Bozkurt, 20.11.2019