31
Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools Last Updated: 2020-02-26

Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team [email protected]

  • Upload
    others

  • View
    17

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

Red Hat OpenStack Platform 13

Monitoring Tools Configuration Guide

A guide to OpenStack logging and monitoring tools

Last Updated: 2020-02-26

Page 2: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com
Page 3: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

A guide to OpenStack logging and monitoring tools

OpenStack [email protected]

Page 4: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

Legal Notice

Copyright © 2020 Red Hat, Inc.

The text of and illustrations in this document are licensed by Red Hat under a Creative CommonsAttribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA isavailable athttp://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you mustprovide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift,Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United Statesand other countries.

Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.

Java ® is a registered trademark of Oracle and/or its affiliates.

XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United Statesand/or other countries.

MySQL ® is a registered trademark of MySQL AB in the United States, the European Union andother countries.

Node.js ® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by theofficial Joyent Node.js open source or commercial project.

The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marksor trademarks/service marks of the OpenStack Foundation, in the United States and othercountries and are used with the OpenStack Foundation's permission. We are not affiliated with,endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Abstract

This guide provides information on configuring logging and monitoring for a Red Hat OpenStackPlatform environment.

Page 5: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table of Contents

CHAPTER 1. INTRODUCTION

CHAPTER 2. ARCHITECTURE2.1. CENTRALIZED LOGGING2.2. AVAILABILITY MONITORING

CHAPTER 3. INSTALL THE CLIENT-SIDE TOOLS3.1. SETTING CENTRALIZED LOGGING CLIENT PARAMETERS3.2. SETTING AVAILABILITY MONITORING CLIENT PARAMETERS3.3. SETTING PERFORMANCE MONITORING CLIENT PARAMETERS3.4. COLLECTD PLUGINS3.5. INSTALLING OPERATIONAL TOOLS ON OVERCLOUD NODES3.6. FILTERING AND TRANSFORMING LOGGING DATA

CHAPTER 4. INSTALLING THE SERVER-SIDE COMPONENTS

CHAPTER 5. MONITOR THE OPENSTACK PLATFORM

CHAPTER 6. VALIDATE THE SENSU CLIENT INSTALLATION

CHAPTER 7. REVIEW THE STATE OF A NODE

CHAPTER 8. REVIEW THE STATE OF AN OPENSTACK SERVICE

APPENDIX A. COLLECTD PLUGINS AND THE PUPPET-BASED CONFIGURATIONS

3

447

111111

12131313

15

16

17

18

19

20

Table of Contents

1

Page 6: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

2

Page 7: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

CHAPTER 1. INTRODUCTIONMonitoring tools are an optional suite of tools designed to help operators maintain an OpenStackenvironment. The tools perform the following functions:

Centralized logging: Allows you gather logs from all components in the OpenStack environmentin one central location. You can identify problems across all nodes and services, and optionally,export the log data to Red Hat for assistance in diagnosing problems.

Availability monitoring: Allows you to monitor all components in the OpenStack environmentand determine if any components are currently experiencing outages or are otherwise notfunctional. You can also configure the system to alert you when problems are identified.

CHAPTER 1. INTRODUCTION

3

Page 8: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

CHAPTER 2. ARCHITECTUREMonitoring tools use a client-server model with the client deployed onto the Red Hat OpenStackPlatform overcloud nodes. The Fluentd service provides client-side centralized logging (CL) and theSensu client service provides client-side availability monitoring (AM).

2.1. CENTRALIZED LOGGING

Centralized logging allows you to have one central place to view logs across your entire OpenStackenvironment. These logs come from the operating system, such as syslog and audit log files,infrastructure components such as RabbitMQ and MariaDB, and OpenStack services such as Identity,Compute, and others.

The centralized logging toolchain consists of a number of components, including:

Log Collection Agent (Fluentd)

Log Relay/Transformer (Fluentd)

Data Store (Elasticsearch)

API/Presentation Layer (Kibana)

NOTE

The director does not deploy the server-side components for centralized logging. RedHat does not support the server-side components, including the Elasticsearch database,Kibana, and Fluentd with plugins running as a log aggregator.

The centralized logging components and their interactions are laid out in the following diagrams:

NOTE

Items shown in blue denote Red Hat-supported components.

Figure 2.1. Centralized logging architecture at a high level

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

4

Page 9: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

Figure 2.1. Centralized logging architecture at a high level

Figure 2.2. Single-node deployment for Red Hat OpenStack Platform

CHAPTER 2. ARCHITECTURE

5

Page 10: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

Figure 2.2. Single-node deployment for Red Hat OpenStack Platform

Figure 2.3. HA deployment for Red Hat OpenStack Platform

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

6

Page 11: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

Figure 2.3. HA deployment for Red Hat OpenStack Platform

2.2. AVAILABILITY MONITORING

Availability monitoring allows you to have one central place to monitor the high-level functionality of allcomponents across your entire OpenStack environment.

The availability monitoring toolchain consists of a number of components, including:

Monitoring Agent (Sensu client)

Monitoring Relay/Proxy (RabbitMQ)

Monitoring Controller/Server (Sensu server)

API/Presentation Layer (Uchiwa)

NOTE

The director does not deploy the server-side components for availability monitoring. RedHat does not support the server-side components, including Uchiwa, Sensu Server, theSensu API plus RabbitMQ, and a Redis instance running on a monitoring node.

The availability monitoring components and their interactions are laid out in the following diagrams:

NOTE

Items shown in blue denote Red Hat-supported components.

CHAPTER 2. ARCHITECTURE

7

Page 12: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

Figure 2.4. Availability monitoring architecture at a high level

Figure 2.5. Single-node deployment for Red Hat OpenStack Platform

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

8

Page 13: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

Figure 2.5. Single-node deployment for Red Hat OpenStack Platform

Figure 2.6. HA deployment for Red Hat OpenStack Platform

CHAPTER 2. ARCHITECTURE

9

Page 14: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

Figure 2.6. HA deployment for Red Hat OpenStack Platform

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

10

Page 15: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

CHAPTER 3. INSTALL THE CLIENT-SIDE TOOLSBefore you deploy the overcloud, you need to determine the configuration settings to apply to eachclient. Copy the example environment files from the director’s heat template collection and modifythem to suit your environment.

3.1. SETTING CENTRALIZED LOGGING CLIENT PARAMETERS

For Fluentd configuration settings, copy /usr/share/openstack-tripleo-heat-templates/environments/logging-environment.yaml and modify the file to suit your environment. Forexample:

Simple configuration

resource_registry: OS::TripleO::Services::Fluentd: ../docker/services/fluentd.yaml

parameter_defaults: LoggingServers: - host: log0.example.com port: 24224 - host: log1.example.com port: 24224

Example SSL configuration

## (note the use of port 24284 for ssl connections)

resource_registry: OS::TripleO::Services::Fluentd: ../docker/services/fluentd.yaml

parameter_defaults: LoggingServers: - host: 192.0.2.11 port: 24284 LoggingUsesSSL: true LoggingSharedKey: secret LoggingSSLCertificate: | -----BEGIN CERTIFICATE----- ...certificate data here... -----END CERTIFICATE-----

LoggingServers - The destination system that receives Fluentd log messages.

LoggingUsesSSL - Determines whether secure_forward is used when forwarding logmessages.

LoggingSharedKey - The shared secret used by secure_forward.

LoggingSSLCertificate - The PEM-encoded contents of the SSL CA certificate.

3.2. SETTING AVAILABILITY MONITORING CLIENT PARAMETERS

For the Sensu client configuration settings, copy /usr/share/openstack-tripleo-heat-

CHAPTER 3. INSTALL THE CLIENT-SIDE TOOLS

11

Page 16: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

For the Sensu client configuration settings, copy /usr/share/openstack-tripleo-heat-templates/environments/monitoring-environment.yaml and modify the file to suit your environment.For example:

resource_registry: OS::TripleO::Services::SensuClient: ../docker/services/sensu-client.yaml

parameter_defaults: MonitoringRabbitHost: 10.10.10.10 MonitoringRabbitPort: 5672 MonitoringRabbitUserName: sensu MonitoringRabbitPassword: sensu MonitoringRabbitUseSSL: true MonitoringRabbitVhost: "/sensu" SensuClientCustomConfig: api: warning: 10 critical: 20

MonitoringRabbit - Connects the Sensu client services to the RabbitMQ instance that runs onthe monitoring server.

MonitoringRabbitUseSSL - Enables SSL for the RabbitMQ client. Uses SSL transport if theprivate key or certificate chain are not specified, as below.

MonitoringRabbitSSLPrivateKey - Defines the path to the private key file or can contain thecontents of that file.

MonitoringRabbitSSLCertChain - Defines the private SSL certificate chain to use.

SensuClientCustomConfig - Specifies additional Sensu client configuration. Defines theOpenStack credentials to be used, including username/password, auth_url, tenant, and region.

3.3. SETTING PERFORMANCE MONITORING CLIENT PARAMETERS

Performance monitoring collects system information periodically and provides a mechanism to store andmonitor the values in a variety of ways using a data collecting agent. Red Hat supports the collectddaemon as a collection agent. This daemon stores the data in a time-series database. One of the RedHat supported databases is called Gnocchi. You can use this stored data to monitor systems, findperformance bottlenecks, and predict future system load.

NOTE

Red Hat OpenStack Platform supports performance monitoring only on the client side.

To configure collectd:

1. Copy the file /usr/share/openstack-tripleo-heat-templates/environments/metrics/collectd-standalone.yaml to your local directory. Open the file, set the following parameters, and list theplug-ins you want under CollectdExtraPlugins. You can also provide parameters in the ExtraConfig section:

parameter_defaults: CollectdExtraPlugins: - disk

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

12

Page 17: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

- df - virt

ExtraConfig: collectd::plugin::virt::connection: "qemu:///system" collectd::plugin::virt::hostname_format: "hostname uuid"

By default, collectd comes with the disk, interface, load, memory, processes, and tcpconnsplug-ins. You can add additional plug-ins using the CollectdExtraPlugins parameter. You canalso provide additional configuration information for the CollectdExtraPlugins using the ExtraConfig option as shown. The example above adds the virt plug-in and configures theconnection string and the hostname format.

3.4. COLLECTD PLUGINS

This section describes the configuration possibilities of director, for example, ExtraConfig parameters.To view the collectd plug-ins and configurations, see Appendix A.

3.5. INSTALLING OPERATIONAL TOOLS ON OVERCLOUD NODES

Include the modified YAML files with your openstack overcloud deploy command to install the Sensuclient and Fluentd tools on all overcloud nodes. For example:

$ openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml -e ~/templates/monitoring-environment.yaml -e ~/templates/logging-environment.yaml --control-scale 3 --compute-scale 1 -e ~/templates/collectd-standalone.yaml --ntp-server 192.168.122.10

3.6. FILTERING AND TRANSFORMING LOGGING DATA

You can filter and transform events sent to Fluentd by setting the LoggingDefaultFilters parameter inyour environment file. For example, the record_transformer type can modify incoming events:

parameter_defaults:

LoggingDefaultFilters: - tag_pattern: '**' type: record_transformer enable_ruby: true record: openstack: '{"hostname": "${hostname}","tag": "${tag}","region": "regionOne","inputname":"fluent-plugin-in_tail","name":"fluentd openstack","fluentd version":"0.12.26", "pipeline_metadata": {"collector":{"ipaddr4":"#{ begin Socket.ip_address_list.select { |a| a.ipv4? && !a.ipv4_loopback? }.map { |a| a.ip_address } end}","ipaddr6":"#{ begin Socket.ip_address_list.select { |a| a.ipv6? && !a.ipv6_loopback? }.map { |a| a.ip_address } end}"}}}' remove_keys: host

- tag_pattern: 'openstack.**' type: record_transformer record: service: '${tag_parts[1]}'

- tag_pattern: 'system.messages**'

CHAPTER 3. INSTALL THE CLIENT-SIDE TOOLS

13

Page 18: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

type: record_transformer enable_ruby: true record: openstack: '{"info": "${record[''message'']}", "systemd":{"t":{"PID":"${record[''pid'']}"},"u":{"SYSLOG_IDENTIFIER":"${record[''ident'']}"}}}' remove_keys: 'ident,message,pid'

As a result, the data received by Kibana has been transformed accordingly:

{ "_index": "logstash-2017.06.29", "_type": "fluentd", "_id": "AVz132QmRtyd8nnlv_11", "_score": null, "_source": { "pid": "22691", "priority": "INFO", "message": "cinder.api.openstack.wsgi [req-04bc2808-f86f-4443-86e6-bfc596969937 - - - - -] OPTIONS http://overcloud-controller-0.lab.local/", "openstack": { "hostname": "overcloud-controller-0", "tag": "openstack.cinder.api", "region": "regionOne", "inputname": "fluent-plugin-in_tail", "name": "fluentd openstack", "fluentd version": "0.12.26", "pipeline_metadata": { "collector": { "ipaddr4": "[\"192.168.24.14\", \"192.168.24.8\", \"10.0.0.4\", \"10.0.0.5\", \"172.16.2.8\", \"172.16.2.4\", \"172.16.2.14\", \"172.16.1.7\", \"172.16.1.9\", \"172.16.3.10\", \"172.16.3.11\", \"172.16.0.14\"]", "ipaddr6": "[\"fe80::293:33ff:fed8:2228%eth0\", \"fe80::293:33ff:fed8:2228%br-ex\", \"fe80::b86c:79ff:fe8f:9fb8%vlan10\", \"fe80::4c78:6fff:feff:14fc%vlan20\", \"fe80::9ced:dfff:fe8c:2d62%vlan30\", \"fe80::ecde:1bff:fe5d:e362%vlan40\", \"fe80::549c:51ff:feea:dfa8%vlan50\", \"fe80::e093:8fff:fef9:69b6%vxlan_sys_4789\"]" } } }, "service": "cinder", "@timestamp": "2017-06-29T21:59:38+00:00" }, "fields": { "@timestamp": [ 1498773578000 ] }, "sort": [ 1498773578000 ]}

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

14

Page 19: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

CHAPTER 4. INSTALLING THE SERVER-SIDE COMPONENTS

NOTE

Red Hat does not support the server-side components and their deployment process.Red Hat does not support opstools-ansible; it is used for demo purposes.

You can use the opstools-ansible playbook to install the server-side components onto Red HatEnterprise Linux 7. These server-side components include availability monitoring and centralizedlogging services that complement the Red Hat-supported client-side components. The most tested opstools-ansible scenario is the deployment of server-side components onto CentOS 7. Detailedinstructions can be found in the README.md: https://github.com/centos-opstools/opstools-ansible

CHAPTER 4. INSTALLING THE SERVER-SIDE COMPONENTS

15

Page 20: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

CHAPTER 5. MONITOR THE OPENSTACK PLATFORMSee the Sensu documentation for further details about the Sensu stack infrastructure:https://docs.sensu.io/sensu-core/1.7/overview/architecture/

Red Hat supplies a set of check scripts in the osops-tools-monitoring-oschecks package. The majorityof the check scripts only check the API connection to the OpenStack component. However, certainscripts also perform additional OpenStack resource tests for OpenStack Compute (nova), OpenStackBlock Storage (cinder), OpenStack Image (glance), and OpenStack Networking (neutron). For example,the OpenStack Identity (keystone) API check returns the following result when keystone is running:

OK: Got a token, Keystone API is working.

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

16

Page 21: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

CHAPTER 6. VALIDATE THE SENSU CLIENT INSTALLATION1. Check the status of the sensu-client on each overcloud node:

# docker ps | grep sensu-client

2. Review the error log for any issues: /var/log/containers/sensu/sensu-client.log

3. Verify that each overcloud node has the /var/lib/config-data/puppet-generated/sensu/etc/sensu/conf.d/rabbitmq.json file that sets the IP address of themonitoring server.

CHAPTER 6. VALIDATE THE SENSU CLIENT INSTALLATION

17

Page 22: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

CHAPTER 7. REVIEW THE STATE OF A NODEIf you have a deployment of the Uchiwa dashboard, you can use it with the Sensu server to review thestate of your nodes:

1. Login to the Uchiwa dashboard and click the Data Center tab to confirm that the Data Center isoperational.

http://<SERVER_IP_ADDRESS>/uchiwa

2. Check that all overcloud nodes are in a Connected state.

3. At a suitable time, reboot one of the overcloud nodes and review the rebooted node’s status inthe Uchiwa dashboard. After the reboot completes, verify that the node successfully re-connects to the Sensu server and starts executing checks.

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

18

Page 23: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

CHAPTER 8. REVIEW THE STATE OF AN OPENSTACKSERVICE

This example tests the monitoring of the openstack-ceilometer-central service.

1. Confirm that the openstack-ceilometer-central service is running:

docker ps -a | grep ceilometer

2. Connect to the Uchiwa dashboard and confirm that a successful ceilometer check is presentand running as defined in the ceilometer JSON file.

CHAPTER 8. REVIEW THE STATE OF AN OPENSTACK SERVICE

19

Page 24: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

APPENDIX A. COLLECTD PLUGINS AND THE PUPPET-BASEDCONFIGURATIONS

This section contains a complete list of collectd plugins and configurations.

collectd-aggregation: collectd::plugin::aggregation::aggregators collectd::plugin::aggregation::interval

collectd-battery: collectd::plugin::battery::values_percentage collectd::plugin::battery::report_degraded collectd::plugin::battery::query_state_fs collectd::plugin::battery::interval

collectd-cgroups: collectd::plugin::cgroups::ignore_selected collectd::plugin::cgroups::interval

collectd-conntrack: None

collectd-contextswitch: collectd::plugin::contextswitch::interval

collectd-cpu: collectd::plugin::cpu::reportbystate collectd::plugin::cpu::reportbycpu collectd::plugin::cpu::valuespercentage collectd::plugin::cpu::reportnumcpu collectd::plugin::cpu::reportgueststate collectd::plugin::cpu::subtractgueststate collectd::plugin::cpu::interval

collectd-cpufreq: None

collectd-csv: collectd::plugin::csv::datadir collectd::plugin::csv::storerates collectd::plugin::csv::interval

collectd-df: collectd::plugin::df::devices collectd::plugin::df::fstypes collectd::plugin::df::ignoreselected collectd::plugin::df::mountpoints collectd::plugin::df::reportbydevice collectd::plugin::df::reportinodes collectd::plugin::df::reportreserved collectd::plugin::df::valuesabsolute collectd::plugin::df::valuespercentage collectd::plugin::df::interval

collectd-entropy:

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

20

Page 25: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

collectd::plugin::entropy::interval

collectd-ethstat: collectd::plugin::ethstat::interfaces collectd::plugin::ethstat::maps collectd::plugin::ethstat::mappedonly collectd::plugin::ethstat::interval

collectd-exec: collectd::plugin::exec::commands collectd::plugin::exec::commands_defaults collectd::plugin::exec::globals collectd::plugin::exec::interval

collectd-fhcount: collectd::plugin::fhcount::valuesabsolute collectd::plugin::fhcount::valuespercentage collectd::plugin::fhcount::interval

collectd-filecount: collectd::plugin::filecount::directories collectd::plugin::filecount::interval

collectd-fscache: None

collectd-hddtemp: collectd::plugin::hddtemp::host collectd::plugin::hddtemp::port collectd::plugin::hddtemp::interval

collectd-interface: collectd::plugin::interface::interfaces collectd::plugin::interface::ignoreselected collectd::plugin::interface::reportinactive collectd::plugin::interface::interval

collectd-ipc: None

collectd-irq: collectd::plugin::irq::irqs collectd::plugin::irq::ignoreselected collectd::plugin::irq::interval

collectd-load: collectd::plugin::load::report_relative collectd::plugin::load::interval

collectd-logfile: collectd::plugin::logfile::log_level collectd::plugin::logfile::log_file collectd::plugin::logfile::log_timestamp collectd::plugin::logfile::print_severity collectd::plugin::logfile::interval

APPENDIX A. COLLECTD PLUGINS AND THE PUPPET-BASED CONFIGURATIONS

21

Page 26: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

collectd-memcached: collectd::plugin::memcached::instances collectd::plugin::memcached::interval

collectd-memory: collectd::plugin::memory::valuesabsolute collectd::plugin::memory::valuespercentage collectd::plugin::memory::interval

collectd-network: collectd::plugin::network::timetolive collectd::plugin::network::maxpacketsize collectd::plugin::network::forward collectd::plugin::network::reportstats collectd::plugin::network::listeners collectd::plugin::network::servers collectd::plugin::network::interval

collectd-nfs: collectd::plugin::nfs::interval

collectd-ntpd: collectd::plugin::ntpd::host collectd::plugin::ntpd::port collectd::plugin::ntpd::reverselookups collectd::plugin::ntpd::includeunitid collectd::plugin::ntpd::interval

collectd-numa: None

collectd-openvpn: collectd::plugin::openvpn::statusfile collectd::plugin::openvpn::improvednamingschema collectd::plugin::openvpn::collectcompression collectd::plugin::openvpn::collectindividualusers collectd::plugin::openvpn::collectusercount collectd::plugin::openvpn::interval

collectd-powerdns: collectd::plugin::powerdns::interval collectd::plugin::powerdns::servers collectd::plugin::powerdns::recursors collectd::plugin::powerdns::local_socket collectd::plugin::powerdns::interval

collectd-processes: collectd::plugin::processes::processes collectd::plugin::processes::process_matches collectd::plugin::processes::collect_context_switch collectd::plugin::processes::collect_file_descriptor collectd::plugin::processes::collect_memory_maps collectd::plugin::powerdns::interval

collectd-protocols: collectd::plugin::protocols::ignoreselected

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

22

Page 27: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

collectd::plugin::protocols::values

collectd-statsd: collectd::plugin::statsd::host collectd::plugin::statsd::port collectd::plugin::statsd::deletecounters collectd::plugin::statsd::deletetimers collectd::plugin::statsd::deletegauges collectd::plugin::statsd::deletesets collectd::plugin::statsd::countersum collectd::plugin::statsd::timerpercentile collectd::plugin::statsd::timerlower collectd::plugin::statsd::timerupper collectd::plugin::statsd::timersum collectd::plugin::statsd::timercount collectd::plugin::statsd::interval

collectd-swap: collectd::plugin::swap::reportbydevice collectd::plugin::swap::reportbytes collectd::plugin::swap::valuesabsolute collectd::plugin::swap::valuespercentage collectd::plugin::swap::reportio collectd::plugin::swap::interval

collectd-syslog: collectd::plugin::syslog::log_level collectd::plugin::syslog::notify_level collectd::plugin::syslog::interval

collectd-table: collectd::plugin::table::tables collectd::plugin::table::interval

collectd-tail: collectd::plugin::tail::files collectd::plugin::tail::interval

collectd-tail_csv: collectd::plugin::tail_csv::metrics collectd::plugin::tail_csv::files

collectd-target_v5upgrade: None

collectd-tcpconns: collectd::plugin::tcpconns::localports collectd::plugin::tcpconns::remoteports collectd::plugin::tcpconns::listening collectd::plugin::tcpconns::allportssummary collectd::plugin::tcpconns::interval

collectd-thermal: collectd::plugin::thermal::devices collectd::plugin::thermal::ignoreselected collectd::plugin::thermal::interval

APPENDIX A. COLLECTD PLUGINS AND THE PUPPET-BASED CONFIGURATIONS

23

Page 28: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

collectd-threshold: collectd::plugin::threshold::types collectd::plugin::threshold::plugins collectd::plugin::threshold::hosts collectd::plugin::threshold::interval

collectd-uptime: collectd::plugin::uptime::interval

collectd-users: collectd::plugin::users::interval

collectd-uuid: collectd::plugin::uuid::uuid_file collectd::plugin::uuid::interval

collectd-vmem: collectd::plugin::vmem::verbose collectd::plugin::vmem::interval

collectd-write_graphite: collectd::plugin::write_graphite::carbons collectd::plugin::write_graphite::carbon_defaults collectd::plugin::write_graphite::globals

collectd-write_log: collectd::plugin::write_log::format

collectd-zfs_arc: None

collectd-apache: collectd::plugin::apache::instances (ex.: {'localhost' => {'url' => 'http://localhost/mod_status?auto'}}) collectd::plugin::apache::interval

collectd-bind: collectd::plugin::bind::url collectd::plugin::bind::memorystats collectd::plugin::bind::opcodes collectd::plugin::bind::parsetime collectd::plugin::bind::qtypes collectd::plugin::bind::resolverstats collectd::plugin::bind::serverstats collectd::plugin::bind::zonemaintstats collectd::plugin::bind::views collectd::plugin::bind::interval

collectd-ceph: collectd::plugin::ceph::daemons collectd::plugin::ceph::longrunavglatency collectd::plugin::ceph::convertspecialmetrictypes

collectd-curl: collectd::plugin::curl::pages collectd::plugin::curl::interval

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

24

Page 29: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

collectd-curl_json: collectd::plugin::curl_json::url collectd::plugin::curl_json::instance collectd::plugin::curl_json::keys collectd::plugin::curl_json::host collectd::plugin::curl_json::user collectd::plugin::curl_json::password collectd::plugin::curl_json::digest collectd::plugin::curl_json::verifypeer collectd::plugin::curl_json::verifyhost collectd::plugin::curl_json::cacert collectd::plugin::curl_json::header collectd::plugin::curl_json::post collectd::plugin::curl_json::timeout collectd::plugin::curl_json::interval

collectd-dbi: collectd::plugin::dbi::databases collectd::plugin::dbi::queries collectd::plugin::dbi::interval

collectd-disk: collectd::plugin::disk::disks collectd::plugin::disk::ignoreselected collectd::plugin::disk::udevnameattr collectd::plugin::disk::interval

collectd-dns: collectd::plugin::dns::ignoresource collectd::plugin::dns::interface collectd::plugin::dns::selectnumericquerytypes collectd::plugin::dns::interval

collectd-generic-jmx: collectd::plugin::genericjmx::jvmarg

collectd-ipmi: collectd::plugin::ipmi::ignore_selected collectd::plugin::ipmi::notify_sensor_add collectd::plugin::ipmi::notify_sensor_remove collectd::plugin::ipmi::notify_sensor_not_present collectd::plugin::ipmi::sensors collectd::plugin::ipmi::interval

collectd-mysql: collectd::plugin::mysql::interval

collectd-netlink: collectd::plugin::netlink::interfaces collectd::plugin::netlink::verboseinterfaces collectd::plugin::netlink::qdiscs collectd::plugin::netlink::classes collectd::plugin::netlink::filters collectd::plugin::netlink::ignoreselected collectd::plugin::netlink::interval

APPENDIX A. COLLECTD PLUGINS AND THE PUPPET-BASED CONFIGURATIONS

25

Page 30: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

collectd-openldap: collectd::plugin::openldap::instances collectd::plugin::openldap::interval

collectd-ping: collectd::plugin::ping::hosts collectd::plugin::ping::timeout collectd::plugin::ping::ttl collectd::plugin::ping::source_address collectd::plugin::ping::device collectd::plugin::ping::max_missed collectd::plugin::ping::size collectd::plugin::ping::interval

collectd-sensors: collectd::plugin::sensors::sensorconfigfile collectd::plugin::sensors::sensors collectd::plugin::sensors::ignoreselected collectd::plugin::sensors::interval

collectd-smart: collectd::plugin::smart::disks collectd::plugin::smart::ignoreselected collectd::plugin::smart::interval

collectd-snmp: collectd::plugin::snmp::data collectd::plugin::snmp::hosts collectd::plugin::snmp::interval

collectd-virt: collectd::plugin::virt::connection collectd::plugin::virt::refresh_interval collectd::plugin::virt::domain collectd::plugin::virt::block_device collectd::plugin::virt::interface_device collectd::plugin::virt::ignore_selected collectd::plugin::virt::hostname_format collectd::plugin::virt::interface_format collectd::plugin::virt::extra_stats collectd::plugin::virt::interval

collectd-write_http: collectd::plugin::write_http::nodes collectd::plugin::write_http::urls

collectd-write_kafka: collectd::plugin::write_kafka::kafka_host collectd::plugin::write_kafka::kafka_port collectd::plugin::write_kafka::kafka_hosts collectd::plugin::write_kafka::topics

collectd-write_prometheus: collectd::plugin::write_prometheus::port

Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide

26

Page 31: Red Hat OpenStack Platform 13 · Red Hat OpenStack Platform 13 Monitoring Tools Configuration Guide A guide to OpenStack logging and monitoring tools OpenStack Team rhos-docs@redhat.com

collectd-ovs_events: collectd::plugin::ovs_events::address collectd::plugin::ovs_events::dispatch collectd::plugin::ovs_events::interfaces collectd::plugin::ovs_events::send_notification collectd::plugin::ovs_events::$port collectd::plugin::ovs_events::socket

collectd-ovs_stats: collectd::plugin::ovs_stats::address collectd::plugin::ovs_stats::bridges collectd::plugin::ovs_stats::port collectd::plugin::ovs_stats::socket

collectd-connectivity: collectd::plugin::connectivity::interfaces

collectd-procevent: collectd::plugin::procevent::process collectd::plugin::procevent::regex_process collectd::plugin::procevent::buffer_length

collectd-sysevent: collectd::plugin::sysevent::listen_host collectd::plugin::sysevent::listen_port collectd::plugin::sysevent::regex_filter collectd::plugin::sysevent::buffer_size collectd::plugin::sysevent::buffer_length

collectd-iptables: collectd::plugin::iptables::chains collectd::plugin::iptables::chains6 collectd::plugin::iptables::interval

collectd-hugepages: collectd::plugin::hugepages::report_per_node_hp collectd::plugin::hugepages::report_root_hp collectd::plugin::hugepages::values_pages collectd::plugin::hugepages::values_bytes collectd::plugin::hugepages::values_percentage collectd::plugin::hugepages::interval

collectd-turbostat: collectd::plugin::turbostat::core_c_states collectd::plugin::turbostat::package_c_states collectd::plugin::turbostat::system_management_interrupt collectd::plugin::turbostat::digital_temperature_sensor collectd::plugin::turbostat::tcc_activation_temp collectd::plugin::turbostat::running_average_power_limit collectd::plugin::turbostat::logical_core_names

APPENDIX A. COLLECTD PLUGINS AND THE PUPPET-BASED CONFIGURATIONS

27