Upload
vmworld
View
93
Download
5
Tags:
Embed Size (px)
DESCRIPTION
VMworld 2013 Michael White, VMware Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Citation preview
Tips and Tricks with vCenter Log Insight (NEW!)
Michael White, VMware
VCM4528
#VCM4528
2
Problem: Operate and Troubleshoot a Complex System
VMware Logs
OS and
App Logs
200 ESXi Host + VMs = 200GB or 2B log events per day
Physical Infrastructure Logs
3
4
Introducing VMware vCenter Log Insight
VMware’s New Log Analytics Solution
• Make sense of all your log data
• Best for vSphere logs, extensible to OS, app,
storage and networking device logs
• Easy-to-use virtual appliance
• Simple and predictable pricing model
Key Use Cases
• IT Operations – Troubleshooting, Monitoring,
Root Cause Analysis
• Security Monitoring, Compliance, Business
Transaction Monitoring, …
Available Now!
• 60-day Trial: www.vmware.com/try-vmware
5
Agenda
Install
Configure
Reporters
Tagging
Content Pack
Scalability
Examples
Demo
Miscellaneous
The End and Thank you!
(Appendix)
6
Install Tidbits
Use FQDN for name during deploy
Before power on, add disk
Add 100 GB to start and figure out what you need (we’ll help)
Have at least one source configured before install
No spelling checker in the Network info area – double check!
Data-core should be what you added + 97GB – this is storage
for events
7
Configure
Once installed, we need to configure for use
Before you start configuring, change root password at console –
this will enable SSH support.
8
Configure – Continued
Now connect to the vC Log Insight URL
9
Configure – Continued
10
Configure – Continued
Add your license and use the Set Key button
11
Configure – Continued
12
Configure – Continued
13
Configure – Continued
14
Configure – Continued
15
Configure – Continued
16
Configure – Continued
17
Configure – Continued
18
Sources
Whole stack is key!
Storage – some easier than others
Networking – Cisco, vCNS – both easy
ESX(i) – easy
vCenter (vC) – harder
vCenter Server Appliance (vCSA) – easy but with a catch
View – can send only events but not not anything else – so treat
like Windows vC
Things to know
• Links in Appendix
• ESXi stops reporting when interrupted – needs attention
19
Sources – Continued
Things to know – Continued
• Windows is harder – need to use a forwarder – I use Datagram
• When using a forwarder log location is key – Check Appendix for locations
20
Sources – Continued
21
Tagging
Important for when you have one host or VM with many log files
being sent to LI
Doing a search will normally search all of the log files from a host
If you use tagging, you can do a search on host AND tag, and
assuming one tag per log file you can do a much more granular
search which is quicker and more applicable
22
Tagging – Continued – No Tagging – on a vCSA
# vpxd source log
source vpxd {
file("/var/log/vmware/vpx/vpxd.log" follow_freq(1) flags(no-parse));
file("/var/log/vmware/vpx/vpxd-alert.log" follow_freq(1) flags(no-parse));
file("/var/log/vmware/vpx/vws.log" follow_freq(1) flags(no-parse));
file("/var/log/vmware/vpx/vmware-vpxd.log" follow_freq(1) flags(no-parse));
file("/var/log/vmware/vpx/inventoryservice/ds.log" follow_freq(1) flags(no-parse));
};
# Remote Syslog Host
destination remote_syslog {
udp("a.b.c.d" port (514));
;
# Log vCenter Server vpxd log remotely
log {
source(vpxd);
destination(remote_syslog);
};
23
Tagging – Continued
So using the tags looks like:
file("/var/log/vmware/vpx/vpxd.log" follow_freq(1) log_prefix(“VC_APP: “) flags(no-parse));
file("/var/log/vmware/vpx/vpxd-alert.log" follow_freq(1) log_prefix(“VC_ALERT: “) flags(no-parse));
file("/var/log/vmware/vpx/vws.log" follow_freq(1) log_prefix(“VC_VWS: “) flags(no-parse));
file("/var/log/vmware/vpx/vmware-vpxd.log" follow_freq(1) log_prefix(“VC_VMW_VPX: “) flags(no-parse));
file("/var/log/vmware/vpx/inventoryservice/ds.log" follow_freq(1) log_prefix(“VC_IS: “) flags(no-parse));
24
Tagging – Continued – Normal
So when using the tags to search looks like:
;
25
Content Packs
A Content Pack provides best practices and
knowledge about the logs
It consists of: Queries, alerts, dashboards
and field extractions
VMware and our partners are working on
Content Packs
vSphere Content Pack
• Ships out of the box
• Knowledge about ESXi and vCenter Server logs as
well as vC Alarms, Events & Tasks
• It consists of: Queries, alerts, dashboards and field
extractions
• Divided into functional categories
• ESX, Storage and vCenter including Alarms
• vSphere and Content Pack dashboards are NOT
editable – users can clone them into their workspace
26
Content Packs – Continued
27
Content Packs – Continued
28
Content Packs – Continued
29
Announcing the Log Insight Content Pack Market Place
And more…
https://solutionexchange.vmware.com/store/loginsight
Extend vCenter Log Insight with Content Packs from:
30
Scalability – Guidelines
Watch ‘outside’ of VM with your normal tools, i.e vC
Operations Manager
Watch ‘inside of vC LI with Health \ System Info
31
Scalability – Guidelines – Continued
32
Scalability – Guidelines – Continued
33
Scalability – Guidelines – Storage
In case we misjudge on storage, enable Data Archiving
Remember that events, once in vC LI are rotated out as disk space
usable is reduced – either to trash or Data Archiving (system alert)
– first in, and first out
If you have to import archived events, than use new instance of LI!
Rough guide – 250 MB per day per ESX host, and 50 MB per day
for other devices – retention time is decided by available storage
and archiving
34
Scalability – Guidelines – Storage
You can enable Data Archiving on the Storage window in
Administration. Once enabled you will be alerted when Archiving
is about to occur. At that time can add disk or not!
35
How Much Disk Space for 30 Days Retention?
Gross estimate:
267 bytes/message
This example:
23*267*60*60*24*30
= ~16GB per
30 days
More accurate estimations can be found in runtime.log
During failures, log volume will increase significantly
• Overprovision!
36
Examples – Bad Credentials
37
Examples – High Latency by Host
38
Miscellaneous
Support Log
• UI – On the Health page of Settings Administration
• CLI – log in on console and execute loginsight-support
• With every support call!
Backup
• VDP, VDPA, etc.
• Image
vC Ops
• Launch in Context
39
Miscellaneous
vC Ops
• Launch in Context
40
Miscellaneous – Continued – Alerts
• vC Ops option requires the
integration enabled and
email requires SMTP
• User alerts are different
from system alerts
• The admin cannot disable
individual alerts
41
Miscellaneous – Continued
Upgrades / Updates
• Will be a short outage
• In-place which makes it easy
• Get .rpm same place on vmware.com you got .ova
• SCP update file to LI in /tmp and execute with rpm –Uvh file_name
• Than test and check Settings \ About for new version – does it match?
42
Miscellaneous – Continued
Fixing IP issues
• Not too hard but tricky – is
best to get it right!
• Install again correctly is
great choice
• vApp modifications is other
choice – make sure VM is off,
and than Edit Settings
– vApp Options
• Not aware of any other
safe alternatives!
43
Demo
44
Summary
Source(s) working first
Add disk at the beginning to avoid outages
Ensure SMTP / vC / vC Operations connections good!
Set a good system email address destination
Monitor disk / processor carefully at first
Use Data Archiving
Most important – make sure your entire stack is reporting
Update as often as you can!
45
Other VMware Activities Related to This Session
HOL: VMware Log Insight
VMware Booth: VMware Cloud Operations
Breakout Session: Deep Dive Wed, 10-11 VCM4445
Group Discussions: Wed, 2-3 Log Insight with Steve Flanders
5 Free License Trial available when you follow @vmLogInsight
HOL:
HOL-SDC-1301
VMware vCenter Log Insight - Unchained from the Allegory
Group Discussions:
VCM1005-GD
Log Insight with Steve Flanders
THANK YOU
Tips and Tricks with vCenter Log Insight (NEW!)
Michael White, VMware
VCM4528
#VCM4528
49
Appendix
Links
• Configuring Remote Syslog on VMware products -
http://sflanders.net/2013/06/24/configure-remote-syslog-on-vmware-products/
• Datagram - http://www.syslogserver.com/syslogagent.html
• Release notes - http://www.vmware.com/support/log-insight/doc/log-insight-10-
release-notes.html
• NetApp syslog - https://communities.netapp.com/docs/DOC-5048
• vCloud Suite - http://www.virtuallyghetto.com/2013/06/forwarding-logs-from-
vcloud-suite-to.html - includes a script to help which include tagging!
• ESXi, syslog, and logins – great blog about how to capture logins – of different
types in ESXi - http://blogs.vmware.com/vsphere/2013/07/capturing-logins-to-
esxi-by-a-root-account.html
• Symmetrix - http://codyhosterman.com/2013/07/10/using-vmwares-vcenter-log-
insight-with-symmetrix-vmax/
50
Appendix – Continued
Links – Continued
• Detecting stopped ESXi syslog forwarding -
http://www.virtuallyghetto.com/2012/07/detecting-esxi-remote-syslog-
connection.html - important, and I suggest using script option
• VM Monitoring log forwarding - http://www.virtuallyghetto.com/2013/07/a-hidden-
vsphere-51-gem-forwarding_10.html
• Install and Admin Guide - http://www.vmware.com/pdf/log-insight-10-install-
admin-guide.pdf
• Users Guide - http://www.vmware.com/pdf/log-insight-10-users-guide.pdf
• Security Guide - http://www.vmware.com/pdf/log-insight-10-security-guide.pdf
• Sample for firewall - http://www.virtualclouds.co.za/?p=740
• Sending Alerts to vC Ops - http://www.virtualclouds.co.za/?p=771
• Location of log files for VMware products – http://kb.vmware.com/kb/1021806
• LI community - http://loginsight.vmware.com
• Try it out - http://www.vmware.com/go/try-log-insight
51
Architecture Overview: Log Insight Deployment Option 1
Considerations:
• Good for log
management greenfield
• Less flexible as syslog-ng
can split the logs into
multiple destinations (e.g.
one to syslog one to local
disk) but LI cannot. Some
senders might still be able to
split reporting
• One UI for everything!
• Easy
ESXi
#1
ESXi
#2 … ESXi
#n
No syslog-ng/rsyslog
Log
Insight
Windows
Epilog or Datagram Syslog
Agent for file-to-syslog
52
Architecture Overview: Log Insight Deployment Option 2
Considerations:
• Requires managing
another syslog server
• More flexible as syslog-ng
can split the logs into
multiple destinations (e.g.
one to syslog one to
local disk)
• For large installations can be
more scalable as you can
have multiple levels of
rollups (e.g. one for each
“pod” or datacenter)
ESXi
#1
ESXi
#2 … ESXi
#n
Syslog
relay
Using a syslog-ng/rsyslog relay
Log
Insight
Windows
Epilog or Datagram Syslog
Agent for file-to-syslog
53
Appendix – Continued
Install Outline
Working in vSphere Web Client
54
Appendix – Continued
Install Outline – Continued
vSphere Web Client doesn’t see .ova by default (.ovl) so you need
to switch to see it – should be different soon – maybe!
55
Appendix – Continued
Install Outline – Continued
Most Important – use fully qualified domain name!
56
Appendix – Continued
Install Outline – Continued
Make sure to have enough space for now, and room to grow!
57
Appendix – Continued
Install Outline – Continued
No spelling checker here – get it all right!!
58
Appendix – Continued
Install Outline – Continued
No power on, as we need to adjust disk to start
59
Appendix – Continued
Install Outline – Continued
60
Appendix – Continued
Install Outline - Continued
61
Appendix – Continued
Install Outline – finished!