Octavia Hands-On Lab

Adam HarwellCarlos GarzaFranklin Naval

German EichbergerMichael JohnsonAl Miller

Susanne Balle Stephen Balukoff

Are you ready?Grab a USB key or download from http://tinyurl.com/octaviaWorkshop-tar-bz

A computer with

8+ GB of RAM

VMware Workstation/Fusion - trial versions of both included on the USB drive

Windows users need ssh client (putty) or work in the vmware window

Copy OctaviaWorkshop from USB drive

Unzip with tar –jxvf or 7zip on windows

Start VMWare - make sure VT-X and nested Virtualization is enabled

VM also at ...

VMWare Fusion Setup (VT-X Setting) Mac

Agenda

• Introduction and Architecture

• Operations

• Troubleshooting

• How to contribute

• Q&A

Introduction and Architecture

Octavia High Level Architecture

Operations

Ground rules• Slide: https://goo.gl/O9c5jn• To login User: ubuntu Pwd: ubuntu• sudo su - stack• cd /opt/stack/devstack/• . openrc admin admin• script /dev/null ← This will fix the devstack’s

screen problem• now you are good to go

Create a load balancer• The vm contains two web servers on port 80, so as user admin:

• nova list - take note of the two ips

• LB: neutron lbaas-loadbalancer-create --name lb1 private-subnet

• Check if ACTIVE with neutron lbaas-loadbalancer-list - take note of VIP

• Listener: neutron lbaas-listener-create --loadbalancer lb1 --protocol HTTP --protocol-port 80 --name listener1

• Pool: neutron lbaas-pool-create --lb-algorithm ROUND_ROBIN --listener listener1 --protocol HTTP --name pool1

• For each member: neutron lbaas-member-create --subnet private-subnet --address <ip> --protocol-port 80 pool1

• Test: curl <vip>

Get information about amphora - 1As admin:

Show all amphora vms: nova list --name amphora

Amphora info commands:curl -k --cert /etc/octavia/certs/client.pem https://<ip on lb-mgmt-net>:9443/0.5/info

curl -k --cert /etc/octavia/certs/client.pem https://<ip on lb-mgmt-net>:9443/0.5/details

curl -k --cert /etc/octavia/certs/client.pem https://<ip on lb-mgmt-net>:9443/0.5/listeners

curl -k --cert /etc/octavia/certs/client.pem https://<ip on lb-mgmt-net>:9443/0.5/listeners/ <listener-id>

Get information about amphora - 2

As admin:

• Log in to a VM: • ssh -i /etc/octavia/.ssh/octavia_ssh_key ubuntu@<ip on lb-mgmt-net> --

good for debugging amphora changes

• Check on agent: sudo service amphora-agent status

• Check on haproxy: sudo service haproxy-<listener-id> status

Force/Simulate a fail over -1• Set up health monitoring:

• Edit /etc/octavia/octavia.conf

• Uncomment controller_ip_port_list and set to controller_ip_port_list = 172.24.4.1:5555 (this is some voodoo due to some bug in devstack we still need to fix)

• Restart o-cw:

• ./rejoin-stack.sh

• find o-cw screen; ctrl-c; cursor up; return

• find o-hm screen; ctrl-c; cursor up; return

• ctrl-a d

• Create another loadbalancer (see the other slide) – make sure to use different names, e.g. lb2, listener2, etc.

Force/Simulate a fail over - 2

• Perform a failover:

• ssh into the amphora: ssh -i /etc/octavia/.ssh/octavia_ssh_key ubuntu@<ip on lb-mgmt-net>

• Stop the amphora agent: sudo service amphora-agent stop

• Observe the fail over:

• in one window curl the vip: while :; do curl <vip>; done;

• in the other one monitor nova (repeat until ip changes): nova list --name amphora

Troubleshooting

Increase Log Level• Edit /etc/octavia/octavia.conf

• Uncomment debug = False and replace False with true. Alternatively you can also uncomment verbose=False and replace with True if you only require INFO level logging. We recommend DEBUG in most cases.

• Restart o-cw, o-api, o-*:• ./rejoin-stack.sh

• find o-* screen; ctrl-c; cursor up; return

• repeat until all services you like to debug are restarted

• ctrl-a d

Log files on the host• Review /opt/stack/logs

• ls /opt/stack/logs/o-*.log

• o-api.log o-cw.log o-hk.log o-hm.log

• o-api.log - logs the api server. Useful to see if requests made it to the system and if they actually got executed

• o-cw.log - logs the controller worker. This does most of the work and this logs will usually aid in debugging problems with nova, neutron, etc.

• o-hk.log - logs the housekeeping manager. Here we do the deletion of amphora marked for deletion, certificate rotation (the ones on the amphora), etc.

• o-hm.log - logs the information about the healthmanager and heartbeats the amphorae send; this is good for failover debugging

Log files on the amphora

• Ssh into the amphora

• cd /var/log/upstart/

• sudo sudo less amphora-agent.log• Useful to review if the amphora is sending heartbeats and if the

API commands were successful; essential for contoller-worker <--> amphora communication debugging

Octavia Database• mysql

• use octavia

• show tables;• There is some interesting stuff but let’s just look at

• Amphora table - information about all the amphora in the system

• Load_balancer, Listener, Member, Pool, Health_monitor - what you expect

• amphora_health - this has a timestamp of the last time we received a heartbeat from the amphora

How to contribute?For more information:

• Our next presentation “LBaaS V2 - Liberty and beyond” – 2:40 Thursday

• Weekly meetings Wednesdays, 20.00 UTC

• Freenode IRC: #openstack-lbaas

• https://wiki.openstack.org/wiki/octavia

• http://www.octavia.io

• https://launchpad.net/octavia

• https://github.com/openstack/octavia

Q&A