Upload
tyler-britten
View
522
Download
0
Embed Size (px)
Citation preview
* Blue box, an ibm company 1
What does it take?
Maintaining a Healthy OpenStack Cloud
Tyler Britten @vmtyler
* Blue box, an ibm company
Who is blue box?
• In Business Since 2003 Deep operational expertise
• Global Data Center Deployments Now including IBM SoftLayer data centers
• 300+ Blue Box Customers Trusted private infrastructure deployments
• Blue Box Panel Allows Blue Box Cloud to be managed over multiple environments from one NOC
• Acquired by IBM in June of 2015
* Blue box, an ibm company 3
Private Cloud as a service – The best of both worlds
§ Fast to deploy § Scalable § Easy to use § Elastic capabilities
§ Irregular performance § Vendor lock-in
PUBLIC CLOUD
Hosted PCaaS § Fast and flexible § Predictable performance § Dedicated environment § Deployed ‘as a Service’ § Security controls § Private line, VPN or
Cross-connect
§ Security controls § Cost control § Open standards § Integrate to existing IT
§ Challenging to scale § Capital and staff
intensive
PRIVATE CLOUD
+
+
All of the security, control, and predictability of Private Cloud with the ease of Public Cloud
* Blue box, an ibm company
LEADING SECURITY
RICHEST PARTNER ECOSYSTEM
Why Customers Choose Blue Box
True Private Cloud as easy to consume as Public Cloud
* Blue box, an ibm company 5
* Blue box, an ibm company 6
* Blue box, an ibm company
Are you doing what you're supposed to be doing?
7
* Blue box, an ibm company
If not, what happened?
8
* Blue box, an ibm company 9
* Blue box, an ibm company 10
What should I be monitoring?
• The Basics • OpenStack Services • RabbitMQ
* Blue box, an ibm company 11
Monitoring The Basics
• CPU • Memory • Disk Space • RAID & disk issues, fstab • Large Files • NTP, kernel options, syslog
* Blue box, an ibm company 12
Monitoring OpenStack Services
• All OpenStack Services • Don't forget dependent services! • Percona • HAProxy
• Nova Oversubscription • Watch for TMI- Too Many Instances
* Blue box, an ibm company 13
Monitoring RabbitMQ
• Rabbit Services • Rabbit Queues themselves • Specifically Growing without Draining
* Blue box, an ibm company 14
* Blue box, an ibm company 15
Logging Rules
1st Rule: Don't impact the system 2nd Rule: DON'T IMPACT THE SYSTEM
3rd Rule: Don't lose logs
* Blue box, an ibm company 16
How do we do that?
• Avoid dependencies on remote syslog within OpenStack • Python 2.7 SysLogHandler bug in Ubuntu (1081022)
• Log Locally to disk and use Logstash-forwarder (Lumberjack) • Build your logging infrastructure to scale out • Log Alerting - Less is more
* Blue box, an ibm company 17
* Blue box, an ibm company 18
Monitoring tools
• Sensu for monitoring • Graphite / Grafana for
visualization • Pagerduty for alerting • Sensu Plugins • Bash, Python, & Ruby • Basics • Available on Blue Box
Github!
* Blue box, an ibm company 19
Logging Tools
• ELK Stack (Elastic Search, Logstash, Kibana) • Archiving logs older than 30
days to secondary ELK stack • Scaling Out • DNS Round Robin for Load
Balancing • Tag logs appropriately when
forwarding - otherwise stack traces can get impossible
* Blue box, an ibm company 20
* Blue box, an ibm company 21
Ursula
• A tool created by Blue Box for installing, managing, and maintaining OpenStack powered clouds • Leverages Ansible Playbooks • Available on GitHub under the MIT license
• Configuring Logging and Monitoring is part of the playbooks!
* Blue box, an ibm company 22
references
• https://github.com/blueboxgroup/ursula • https://github.com/blueboxgroup/ursula-monitoring/ • https://bugs.launchpad.net/ubuntu/+source/
python2.7/+bug/1081022
* Blue box, an ibm company
THANK YOU FOR YOUR TIME!
twitter.com/bluebox linkedin.com/company/blue-box-group
REACH US AT 119 Pine Street, Suite 200 Seattle, Washington 98101
International: +1.206.607.0660 Fax: 253.272.3005 Mail: [email protected] Web: www.bluebox.net
WHERE WE ARE FIND US AT
23