Large Scale Cluster Management with Ansiblepages.aerospike.com/rs/229-XUE-318/images/The-Trade-Desk...Inside this command, you can optionally specify what the target cluster size is–but

Large Scale Cluster Management

with Ansible

Albert Autin

Data Operations Engineer

The Trade Desk

2 A E R O S P I K E S U M M I T ‘ 1 9 | Proprietary & Confidential | All rights reserved. © 2019 Aerospike Inc

Large Scale Cluster

Management with Ansible

By Albert Autin

Data Operations Engineer at The Trade Desk


▪ My name is Albert Autin, I’m a Data Operations

Engineer at theTradeDesk.

▪ Working at theTradeDesk 1 year.

▪ 4 years working with Aerospike.

▪ I’m active on the Aerospike forums a lot,

@Alb0t, come join us! discuss.aerospike.com

Intro


⁄

▪ Aerospike at theTradeDesk.

▪ What is Aerospike maintenance?

▪ Migrations and maintenance pitfalls.

▪ The cost and risk of maintenance.

▪ Maintenance as code using Ansible.

Agenda


Publishers Ad Exchange Advertisers and Brands

The Crux of (or Intro to) AdTech


▪ 17 clusters, 10 datacenters.

▪ 400B-800B objects, usually 2-5kb in size.

▪ Disk: 3PB+ of NVMe storage

▪ Memory: 140TB+.

▪ 5-12MM reads/s average, 2-5MM writes/s average.

▪ Over 500 production Aerospike servers

▪ Some clusters have 90-115 nodes.

▪ Cloud Deployments.

▪ On-Prem deployments.

▪ Almost all bare-metal.

Global Aerospike footprint at TTD


▪ Decommissioning servers.

▪ Replacing servers.

▪ Upgrading servers.

▪ Updating Aerospike.

▪ Apply kernel updates.

Aerospike Maintenance tasks


Migrations!


Node D

What are migrations – node loss

Node A Node B Node C

X

X Y

Z

Y

Z

Q

Q

Y

Z


Node DNew node!

What are migrations – node add

Node A Node B Node C

X

XY

Z

Y

Z

Q

Q


▪ Storage format changes.

▪ No persistence.

▪ New or wiped drives.

Sooner or later, or sometimes all the time,

you will need to wait for migrations during

maintenance.

When to wait for migrations during maintenance


▪ While the cluster has migrations, it can be in

a vulnerable state.

▪ Quiesce alleviates some of this risk.

▪ The human risk.

▪ Migrations are time consuming, and require

effort to monitor and coordinate.

▪ Considerable resource tax on the system.

Migrations – what’s the big deal?



▪ Each node has their own set of migration tunings.

▪ The reason Aerospike throttles migrations is to keep available resources for business activity.

You don’t want your database to become unavailable while data is moving.

▪ Migration tunings:

▪ migrate-threads: default 1 – control the number of threads used for outgoing migrations.

▪ migrate-max-num-incoming: default 4 – the maximum number of incoming partitions

that can be in progress at once for a particular node.

▪ migrate-sleep: default 1 – microseconds to sleep between each record.

▪ channel-bulk-fds: default 2 – sockets to each neighbor node.

▪ channel-bulk-recv-threads:default 4 – threads for processing incoming migrations.

Migrations – throttling up and down


The cost of maintenance

SalaryHourly Rate

Migration Hours per server

# ServersMaintenance

HoursCost of

maintenance

$50,000 $24.04 2 500 1000 $24,038.46

$100,000 $48.08 2 500 1000 $48,076.92

$150,000 $72.12 2 500 1000 $72,115.38

$50,000 $24.04 8 500 4000 $96,153.85

$100,000 $48.08 8 500 4000 $192,307.69

$150,000 $72.12 8 500 4000 $288,461.54


▪ Different ways to do upgrades

▪ Bring the whole thing down

▪ Losing money while system is down

▪ Globally operating DSP, there is no good down time

▪ Risk of systems not coming back

▪ Data inconsistencies (if not using tombstones)

▪ Batches of ~10%

▪ Evictions

▪ Proxies

▪ Risk of systems not coming back

▪ Rolling upgrade

▪ The most preferred method, safest, easiest to roll back in case of an issue.

▪ Very time consuming

Upgrades


▪ Chef for configuration management.

▪ Ansible for task orchestration.

How does TTD manage all this?


▪ Ruby language for libraries and templates, easy to use

▪ Create config files

▪ Detects version, role selected

▪ Create service definition for version selected (ex. Nsup-period is namespace context in >4.5.1.x)

▪ Create network definition and populate mesh address from role members.

▪ Download binary files for installation

(ex. Aerospike-enterprise-4.5.1.5.tgz)

▪ Manage migration tuning times automatically – throttle up during

low traffic times, down during peak times.

▪ Deploy monitoring software, tuned profiles, health-check scripts, hotkey scripts, partitioning

script, and more.

Maintenance at TTD – Chef config management


▪ YML files describing tasks to take.

▪ Ansible is great for task orchestration.

▪ AWX (Ansible Tower) can be used for

scheduling, automation,

and notifications (like slack!).

▪ We use ansible with Aerospike for:

▪ Setting up new systems after chef deploys

all the binary files, config files, etc.

▪ Performing rolling updates.

▪ Decommissioning systems.

▪ Scheduling and managing backups.

Maintenance at TTD – Ansible task management


▪ YAML/YML text following ansible playbook conventions.

▪ Usually begins with overall playbooks and variables.

▪ List of tasks or steps to take.

▪ Example playbook -

What’s in an Ansible playbook?


There exists plenty of automation and methods

of upgrading software, managing installs, and

configuration management.

The challenge with all of these solutions is that we

need a way to tie in the requirement to wait for

migrations.

The requirements for automation of upgrading or decommissioning Aerospike


▪ TTD replaced over 200 physical

systems in Q3/Q4.

▪ We had to ‘gracefully’ remove these

in groups of 3-4

▪ At first this was done manually.

Decommissioning or replacing servers


▪ Specify the hosts you want to

decommission in an inventory file

(or use a “limit in ansible-speak”)

▪ Run the playbook against those hosts.

▪ The playbook will iterate through each

host, perform backup/quiesce/etc.,

to safely shut the node down.

▪ The playbook will wait for migrations

before proceeding to the next node.

▪ Sends notification when done.

Decommissioning steps –the ansible way


▪ Ansible-playbook

playbook/aerospike_upgrade_demo.yml –

execute with serial: 1

1. Check to see if upgrade needed – this makes

it safe to re-run!

2. Check if Aerospike running.

3. Check for migrations.

4. Quiesce the target node, recluster.

5. Wait for migrations.

6. Take the server down.

7. Perform kernel update.

8. Reboot.

9. Start aerospike.

10. Verify node joins, and migrations finish.

11. Proceed to next node.

Upgrade using Ansible


▪ 4 servers running 4.3.1.3

▪ 3 namespaces

▪ “bat” RF=1, 100,000 records

▪ “baz” RF=1, 500,000 records

▪ “test” RF=2, 1,000,000 records

▪ Requires kernel updates

▪ Aerospike-server-enterprise.tgz placed

in /vagrant folder

Demo – Beginning state


26


▪ If any error is detected, the playbook aborts. No

further hosts are touched.

▪ Review the reason for failure, and run it again.

▪ Because we verify the version before touching

the server, it should be safe to re-run with the

same inventory. If the node is already upgraded,

it is skipped.

▪ If a node that is in the list to be upgraded does

not have aerospike running, we can make that

cause the playbook to abort.

What happens if the playbook is aborted or something goes wrong?



▪ AWX is a web-based user interface for Ansible.

▪ Things we can do with AWX

▪ Schedule playbooks to run.

▪ Long-running playbooks do not need to run on a workstation.

▪ Centralized source for playbooks to run.

▪ Centralized source of inventories – including dynamic modules.

▪ Workflow automation – allowing you to ‘chain together’ playbooks.

▪ Reporting.

▪ Credential management.

▪ Delegation – give users specific permissions to invoke some playbooks.

▪ Notification integration – like Slack!

AWX



aerospike_migrations module parameters


▪ Module will be released as part of ansible core

with ansible 2.8+.

▪ If you want to use it before 2.8, it’s easy!

▪ Download aerospike_migrations.py and place at

~/.ansible/plugins/modules/database/aerospike/

▪ In AWX, place the script in the library path.

▪ Install “aerospike” python library on your server.

▪ Run your playbook

▪ Demo code:

https://github.com/Alb0t/aerospike_upgrade_demo

How to use the module?

https://github.com/Alb0t/aerospike_upgrade_demo


Parameter Choices/Defaults Comments

consecutive_good_checks

integer

Default:

3

How many times should the cluster report “no migrations” consecutively before returning

OK back to ansible?

fail_on_cluster_change

boolean

• Choices:no

• yes ←

Fail if the cluster key changes if something else is changing the cluster, we may want to fail

Host string Default: localhost Which host do we use as seed for info connection

local_only boolean / required •Choices:no

•yes

Do you wish to only check for migrations on the local node before returning,

or do you want all nodes in the cluster to finish before returning?

migrate_rx_keys tring Default:

migrate_rx_partitions_remaining

The metric key used to determine if we have rx migrations remaining. Changeable

due to backwards compatibility.

migrate_tx_key string Default:

migrate_tx_partitions_remaining

The metric key used to determine if we have tx migrations remaining.

Changeable due to backwards compatibility.

min_cluster_size integer Default: 1 Check will return bad until cluster size is met or until tries is exhausted

sleep_between_checks integer Default: 60 How long to sleep between each check (seconds).

target_cluster_size integer When all aerospike builds in the cluster are greater than version 4.3, then the cluster-stable info command

will be used. Inside this command, you can optionally specify what the target cluster size is–but it is not

necessary. You can still rely on min_cluster_size if you don't want to use this option.

If this option is specified on a cluster that has at least 1 host <4.3 then it will be ignored until the min version

reaches 4.3.

tries_limit integer Default: 300 How many times do we poll before giving up and failing?


1) Verify “cluster good state”

a) Verify cluster key is the same across all nodes

b) Verify all nodes report at least the minimum cluster size

c) Verify migrates are allowed

2) If we can use cluster-stable, use that.

3) If we cannot use cluster-stable, check the rx/tx key for

each namespace on all nodes. If any node > 1 – report

migrations still in progress.

• If any particular data point is not found, or all these

conditions are not met – the module will report a failure.

• If you write your playbook with serial:1, a failure from

the module should stop the playbook from continuing

further, and can have AWX notify you.

How does it work?


It’s not perfect.

Source: xkcd.com

Documents

Large Scale Cluster Management with Ansiblepages.aerospike.com/rs/229-XUE-318/images/The-Trade-Desk...Inside this command, you can optionally specify what the target cluster size is–but