ArcSight ESM Performance GuideJoe BurkeSolutions Architect

Version 1.3June 2011

2

ContentsDisclaimer..............................................................................................................................................3Introduction............................................................................................................................................3Choosing the right hardware..................................................................................................................4Tuning Red Hat Enterprise Linux and Oracle........................................................................................6Benchmarking insert performance with Bleep.....................................................................................11Viewing OS statistics with kSar...........................................................................................................14ESM performance statistics.................................................................................................................16Event transport performance...............................................................................................................16Console performance..........................................................................................................................16References.......................................................................................................................................... 17

3

Disclaimer It is highly recommended that those planning to follow this guide have an excellent understanding of Linux, Oracle, and ArcSight ESM. The reader assumes all risk in how they choose to use the information presented. Many of the recommendations presented here are not officially supported by ArcSight.

IntroductionThis guide was written for a technical audience to explain how to design, benchmark, and tune ArcSight ESM for performance. It was written based on ESM 5.0.1 with Oracle 11g, but most of the information could also be applied to ESM 4.x and Oracle 10g. The main body of the document is quite focused and expects the reader to already be familiar with most of the concepts presented. However, I’ve also included a references section to help the reader find additional detail for concepts and terminology they need a better understanding of.

4

Choosing the right hardware

StorageThis is the very first section because it is the most important. Storage is the heart of the database and your choices here will impact nearly every other aspect of how ESM performs.

From the Oracle White Paper:Optimizing and Protecting Storage with Oracle Database 11g Release 2

“Traditionally, storage arrays used underneath Oracle Databases have been sized based on the amount of data that needs to be stored in corresponding databases. As disk capacity has increased, and processor cores have become faster, this sizing metric is no longer useful. Instead, storage arrays should be size based on the performance they can deliver to database applications. ““Storage Arrays should be sized on the number of IOPs they deliver, as well as the achievable MB/s that they can deliver.”

Most storage vendors and storage engineers want to tell you that the product they’re using will offer good performance and redundancy while minimizing cost per gigabyte. That’s fine as long as they can meet your requirements with hard numbers.

Every ESM system should be provisioned with a minimum ratio of 1:1 Events Per Second (EPS) to Input/Output Operations Per Second (IOPS). So if all your connectors will send a total of 2,000 EPS to the ESM manager, plan for your storage solution to provide 2,000 IOPS and verify it with ORION before installing the database. Increasing IOPS almost always starts with adding Hard Disk Drives (HDD) or Solid State Drives (SSD) to your storage solution, regardless of how much capacity you need. Depending on how the system is being used, it’s not uncommon to see ratios of 1:3 - 1:10 (EPS:IOPS) in order to ensure all users can analyze their data in a timely manner.

If you compromise on any area of your ESM system, let it be something other than storage! What’s the penalty for using a storage solution that can’t provide sufficient IOPS with low latency?

Active channels that load slowly or won’t load at all Reports that are slow to complete or never complete Trends that fail Partitions fail to compress and/or archive

If you need to create very large volumes, you can try to use LVM2 to create large EXT3 or GFS2 filesystems. The benefits of having more drives behind your mount point may outweigh the overhead of software striping. A stripe size of 512k usually offers the best performance.

5

Oracle ORIONAs mentioned above, ORION should be used to benchmark your storage solution prior to installing the database.

How to use ORION:1. Download ORION from Oracle’s website (See reference section).2. Create a lun file with a list of the devices to be used for ARC_EVENT_INDEX, and

ARC_EVENT_DATA. For example:orion.lun/dev/mapper/ora1

3. Execute the ORION command. THIS WILL ERASE ALL DATA ON THE DEVICES NAMED IN STEP TWO. You’ll need to adjust the “size_small” parameter based on the Oracle db_block_size you plan to use (usually 32). The “cache_size” parameter should be set to whatever amount of cache is installed on your storage solution (in megabytes).

./orion_linux_x86-64 -run advanced -testname orion -write 65 -size_small 32 -cache_size 4096 –duration 120

4. When the ORION command finishes, you will find a file named “orion_<date>_summary.txt”. IOPS will be listed as “Maximum Small IOPS” in that txt file.

Oracle data files

REDO - Put the Oracle redo logs on their own RAID10 group (a group of drives that are not used by anything other than REDO).

UNDO – Put the UNDO datafiles on their own RAID10 group (a group of drives that are not used by anything other than UNDO).

ARC_EVENT_INDEX, ARC_EVENT_DATA – These 2 tablespaces generate a high volume of I/O and perform best when placed on their own RAID group consisting of solid state storage or a very large number of HDDs. Use RAID10 if selecting HDD storage.

Server considerations

As a general rule of thumb, you need 1 manager server processor core for every 1,000 EPS you plan to send to the manager.

Add as much memory for the database server as you can afford. Among other things, this allows Oracle to service queries for recently inserted events from memory instead of from disk. There is research to suggest diminishing returns for anything over 256GB.

Add a crossover cable between the manager and database server. This will isolate the traffic, providing extra security, performance, and ease of troubleshooting. You can even try jumbo frames on these interfaces to see if that further improves performance.

6

Tuning Red Hat Enterprise Linux and Oracle

Red Hat Enterprise Linux (RHEL) 5.5 x86_64 KickstartChoosing all the correct packages to install when building an ESM database server can be time consuming. Here’s a kickstart file that you can use to quickly build a system that is ready for the ESM 5 database or manager installation. Everything for the install is done for you automatically except for the local storage configuration (which it will prompt you for during the install). The root password is “arcsight5”

#Begin rhel55-ks.cfginstallcdromkey --skiplang en_US.UTF-8keyboard usrootpw --iscrypted $1$kiJoDDMg$yJMIx0pcjfOSWpW8YFfms1firewall --disabledauthconfig --enableshadow --enablemd5selinux --disabledtimezone --utc Etc/GMTbootloader --location=mbr

%packages@base@core@development-libs@development-tools@legacy-software-development@legacy-software-support@system-toolsfipscheckfipscheck-develsgpiolibaiolibstdc++-devellibstdc++44-devellibaio-develunixODBC-develgcc44-c++imakegcc44gnutls-utilscompat-dbunixODBCsysstat-amtu-setroubleshoot-audit

7

-firstboot-sendmail-vnc#End rhel55-ks.cfg

Operating system parametersThese parameters can be tuned to improve performance. In an effort to keep this document a reasonable size, a full explanation of all parameters is not provided. For more information, I strongly recommend reading the document “Oracle 10g Server on Red Hat Enterprise Linux 5” in addition to this performance guide (See link in the references section).

/boot/grub/grub.confAdd “ elevator=deadline” to the end of the kernel line

/etc/rc.localecho 1 > /proc/scsi/sg/allow_dio

/etc/security/limits.conforacle hard nproc <512*processes>oracle soft nproc <512*processes>oracle soft nofile <512*processes>oracle hard nofile <512*processes>oracle hard memlock <shmmax/1024>oracle soft memlock <shmmax/1024>

/etc/sysctl.conffs.aio-max-nr=<Typically 1048576-3145728>fs.file-max=<(512*processes)+32000>net.ipv4.ip_local_port_range=<Typically 1024 65000>net.core.rmem_default=<Typically 262144>net.core.rmem_max=<Typically 4194304>net.core.wmem_default=<Typically 262144>net.core.wmem_max=<Typically 262144>kernel.sem=< semmsl = processes+10, semmns=SEMMSL*SEMMNI,semopm=SEMMSL, semmni=142 (recommended value) >kernel.shmall=<shmmax / 4096>kernel.shmmax=<A value in bytes, slightly higher than sga_max_size>vm.nr_hugepages=<shmmax / 2048 / 1024>vm.hugetlb_shm_group=<The numeric group ID for Oracle>

8

Example Operating System ParametersI’ve included this example from one of my smaller lab systems as a reference.

/boot/grub/grub.confkernel /vmlinuz-2.6.18-194.el5 ro root=LABEL=/ elevator=deadline

/etc/rc.localecho 1 > /proc/scsi/sg/allow_dio

/etc/security/limits.conforacle soft memlock 2752512oracle hard memlock 2752512oracle soft nofile 256000oracle hard nofile 256000 oracle soft nproc 256000oracle hard nproc 256000

/etc/sysctl.conffs.aio-max-nr=3145728 fs.file-max=288000 net.ipv4.ip_local_port_range=1024 65000net.core.rmem_default=262144net.core.rmem_max=4194304net.core.wmem_default=262144net.core.wmem_max=262144kernel.sem=510 72420 510 142 kernel.shmall=688128 kernel.shmmax=2818572288vm.nr_hugepages=1344 vm.hugetlb_shm_group=60 # These should be defaults for swap, but I set them here just to be sure what the values are.vm.swappiness=60vm.dirty_background_ratio=10vm.dirty_ratio=40vm.dirty_expire_centisecs=2999vm.dirty_writeback_centisecs=499vm.min_free_kbytes=8192

9

Oracle parametersThis document was written to assume that you want to use huge_pages which requires Automatic Memory Management (AMM) and Automatic Shared Memory Management (ASMM) to be disabled.

How to disable AMM/ASMM for a DB server with 4GB of memory or more (also sets the log_buffer and DIO/AIO)

1. Login to the database server as root2. su – oracle3. sqlplus / as sysdba4. alter system set memory_target=0;5. alter system set sga_target=0;6. alter system set db_cache_size=1280m;7. alter system set shared_pool_size=768m;8. alter system set large_pool_size=16m;9. alter system set java_pool_size=16m;10.alter system set streams_pool_size=16m;11.alter system set log_buffer=16777216 scope=spfile;12.alter system set disk_asynch_io=TRUE scope=spfile;13.alter system set filesystemio_options=SETALL scope=spfile;14.quit15.exit16.service oracle stop17.umount tmpfs (comment it out in fstab as well)18.vm.nr_hugepages=134419.vm.hugetlb_shm_group=6020.oracle hard memlock 275251221.oracle soft memlock 275251222.sysctl –p23.service oracle start

Other Oracle parametersCompatible=11.2.0.1.0Optimizer_dynamic_sampling=4Optimizer_features_enable=11.2.0.2Db_block_size=32768Db_file_multiblock_read_count=16Db_files=2000Db_writer_processes=8Open_cursors=3000Parallel_degree_policy=manualParallel_max_servers=0Processes=500Sga_max_size=2560mpga_aggregate_target=384m

10

Modifying your redo logsUse these steps to modify the location or size of your redo logs (using SQL plus). You’ll want larger redo logs if they’re switching too often.

1. To show the current status of the redo logs.a. Select group#, status from v$log;

2. Here is an example command to add new redo logs of the location+size of your choice.a. Alter database add logfile group 5 (‘/oracle/archive/redo/redo08b.log’) SIZE 4G;

3. Before you can drop the old redo log files you must ensure they are inactive or unused. Use the select statement from step 1a to see what can be dropped.

a. Follow step 3b if you need to change the status of a log file.b. Alter system switch logfile;c. You may need to restart Oracle if you can’t get a log to switch

4. This is an example of a command to drop a log filea. Alter database drop logfile group 1;

Manager tweaksAdding the following lines to your server.properties file may help increase the EPS you can send to the manager.

servletcontainer.jetty311.threadpool.maximum=256agents.threads.max=128

Other ways to tune performanceMore information on these topics can be found in the ArcSight ESM Admin Guide.

Turbo modes Querytuner RegenerateEventStats SetDynamicSampling GatherSystemStats Speeding up partition compression (nologging)

In addition to these topics, I also recommend disabling all rules and trends that you’re not using.

11

Benchmarking insert performance with Bleep Bleep is a utility packaged with the ArcSight ESM manager to measure write/insert performance. The following sections explain how to use it. The results of testing with Bleep will tell you the maximum Events Per Second (EPS) your system can handle in terms of write performance.

Instructions for using Bleep with ESM in Default mode + Self-signed Cert

1. Stop the partition archiver service and the manager service.DB Server# service arc_oraclepartitionarchiver_db stopManager Server# service arcsight_manager stop

2. Tar and gzip the manager directoryManager Server# cd /home/arcsight/arcsightManager Server# tar –cf bleep.tar Manager/*Manager Server# gzip bleep.tar

3. Transfer bleep.tar.gz from the manager server to the server that will be running Bleep (I usually put it in /root and then run Bleep as root)

Bleep Server# tar –xzf bleep.tar.gzBleep Server# cd /root/Manager/utilities/bleepBleep Server# unzip baseline.events.zip

4. Create a bleep.properties file in the /root/Manager/config directory with the following parameters:

manager.hostname=<manager hostname>manager.user=adminmanager.password=<password>agent.replay.instances=5agent.replay.config.epslimit=-1agent.replay.config.filenames=/root/Manager/utilities/bleep/halloween.final.eventsagent.replay.config.cache.enabled=trueagent.replay.config.cache.maxsize=10000

5. Perform the following checks before running Bleep:a. Ensure that you can ping the manager by it’s hostname from the server running Bleep.b. Start the arcsight_manager service on the Manager Server.c. Make sure the partition archiver is still not running on the DB serverd. Make sure no other connectors are active (Ctrl+Alt+E from the console to check)e. Make sure the web service is stopped if installedf. Make sure all users are logged out of the console

6. Run BleepBleep Server# cd /root/Manager/binBleep Server#./arcsight bleep

You may get some errors when running Bleep, but as long as you’re seeing Avg Total values that change, then it’s ok. Press ctrl+c to stop the test when finished. You’ll want to let the test run for 30 minutes to ensure the insert rate stabilizes. Record the Avg total at the 30 minute mark.

12

Bleep successfully sending events to the manager

You may notice the word “ASS” appear during various stages of testing. This is an acronym for “ArcSight System Stressor”.

You’ll want to reinitialize the database after running Bleep to clear out all the test events. To do this, follow these steps.

1. DB Server# cd /usr/local/arcsight/db/bin2. DB Server# ./rundatabasesetup.sh –i console -w3. Choose option “Initialize ArcSight Tablespaces, Schema and Resources”4. Choose option “Create/Recreate Schema Only”5. Choose option “Provide the host name, port and instance name”6. The rest of the prompts should already be filled in and you can simply hit enter at each prompt.7. Type yes when prompted to delete all objects owned by the ArcSight database user.8. Specify your desired retention period when prompted along with your email notification

information.9. Specify your partition management runtimes10.Hit enter to continue after the ArcSight database has been created and initialized11.You’ll now need to reconfigure the manager and reconnect it to the database12.Manager Server# cd /home/arcsight/arcsight/Manager/bin13.Manager Server# ./arcsight managersetup (Follow the prompts, similar to a new manager

install)

Sometimes Bleep will error out when trying to register itself to the manager. To fix this issue, try adding the following line to server.properties on the manager server.

auth.agent.default-group=/All Connectors/All Agents/Site Agents

13

Additional steps for running Bleep in FIPS mode (for the server running Bleep)

1. On the Bleep server, edit the /root/Manager/bin/scripts/bleep.sh script and add –Dhttps.protocols=TLSv1 to the ARCSIGHT_JVM_OPTIONS section.The entire line would then look like:ARCSIGHT_JVM_OPTIONS=”-Dhttps.protocols=TLSv1 –Xms256m –Xmx1024m –verbose:gc”

2. Launch the keytoolgui utility from the Bleep server, choose “Open KeyStore” and open the cacerts file from the /root/Manager/jre/lib/security folder. The password when prompted is “changeit”, without quotes. To launch the keytool gui from the Bleep server:

a. # startxb. # cd /root/Manager/binc. # ./arcsight keytoolgui

3. Press ctrl+t and locate your manager’s certificate to import (from the “Import Trusted Certificate” dialogue box). The file should have a “crt” extension. The easiest way to get your manager’s certificate is to open https://managerhostname:8443 from Firefox and export the certificate to a file. After clicking the import button, click ok, ok, yes, ok, ok. Then press ctrl+s. Exit the keytoolgui.

Additional step for running Bleep with a demo cert (for the server running Bleep)

1. # cd /root/Manager/bin2. # ./arcsight tempca -ac

Tips for using the Bleep utility with large ESM deployments

For larger systems, the typical settings for Bleep aren’t going to let you reach the full speed your system is capable of. Here are some tips to help.

Change the following parameter in /root/Manager/config/bleep.properties on the Bleep server to a value of 20 or greater (30 has worked well in the past)agent.replay.instances

Change the following parameter in /root/Manager/config/bleep.properties on the Bleep server.agent.replay.config.cache.maxsize=90000

Edit the /root/Manager/bin/scripts/bleep.sh script, and add “–Xms16384m –Xmx16384m –verbose:gc” (without quotes) to the ARCSIGHT_JVM_OPTIONS section. This will allow Bleep to use more memory. This example requires you to have 16GB of physical memory free on the server running Bleep. I also recommend setting your manager to use 16GB.

14

Viewing OS statistics with kSar Tracking performance statistics over time could show you how much a hardware upgrade or configuration change improved performance (by comparing before/after). It can also help diagnose performance issues and allow you to pinpoint the exact moment something changed and how it impacted the system. You can also analyze the data to determine if there are certain times where the system is over-utilized and accordingly make adjustments to the time scheduled reports run, change partition management run times, etc.

Before you start using kSar you must first install the sysstat RPM (which is listed as a required package in the ESM installation guide, so you probably already have it) on your database server which will then automatically gather system statistics via sar.

Disk statistics are not collected by default, but it’s easy to enable it. To enable disk statistics collection:Just add a “-d” in two files, then restart sysstat

vi /etc/init.d/sysstat/usr/lib64/sa/sadc –F –L –d - && touch /tmp/sysstat.run

vi /etc/cron.d/sysstat*/10 * * * * root /usr/lib64/sa/sa1 –d 1 1

Once you have that step completed, head over to SF and grab the latest version of kSar (this doc references v5.0.6).http://sourceforge.net/projects/ksar/

Start the interface using “java –jar ksar.jar”

1. Once open, you’ll need to connect to your server via SSH to grab some data. (There is also an option to load data from a text file)

2. Follow the prompts until you get to the SSH Command window.3. sar –A –f /var/log/sa/sa<day number>4. You now have a wealth of data available for viewing.5. Typical items of interest include

a. Swap statisticsb. Disk Statisticsc. Processor statisticsd. Network Interface statistics

6. You can even create your own graph to compare two different stats using the options -> “add personal graph”.

7. You can also zoom in or change the time range you’re viewing by clicking the “select time range” function.

8. To get the actual disk device name, you’ll need to do “cat /proc/partitions”. Use “Options -> disk name” to assign an easy to recognize alias to any of the disks.

15

16

ESM performance statistics Here are a few interesting events you can track from the ESM console

Track when an active channel is launched or a report is runDeviceEventClassID=channel:001 or report:104

Track DB Insert time (microseconds)DeviceEventCategory=/Monitor/EventBroker/InsertTime

Track report completion time (milliseconds)DeviceEventClassID=report:100

Event transport performance

Increasing the amount of memory used by the connector can help improve performance for connectors that process a high event rate.

agent.wrapper.confMaxNewSize=256mwrapper.java.initmemory=1400wrapper.java.maxmemory=1400

These settings can be tweaked to help sustain high event throughput over a high latency network.

agent.propertiestransport.loggersecure.threads=4http.transport.multithreaded=truehttp.transport.threadcount=4

/etc/sysctl.confnet.core.rmem_max = 16777216net.core.wmem_max = 16777216net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216

net.ipv4.tcp_fin_timeout = 25net.core.netdev_max_backlog = 30000net.ipv4.tcp_tw_recycle = 0net.ipv4.tcp_tw_reuse = 0

/etc/rc.localifconfig eth1 txqueuelen 7500

17

Console performance Here are a few tips and tweaks for the ESM console.

Put the machine running the console close to the manager to minimize network latency. Too much network latency will result in an unresponsive GUI, and delayed loading every time the user scrolls in an active channel. For example, 300ms of network latency can create 5 second delays every time the user scrolls down in an active channel.

Increase the amount of memory used by the consoleEdit ../current/bin/scripts/console.bat and change the following:-Xms1024m-Xmx1024m

You can go higher on console memory, but I haven’t seen any benefit.

References In addition to the following links, I also strongly recommend you read the ESM install guide, ESM admin guide, and ESM 101 for the version of ESM that you’re using.

Red Hat Enterprise Linux

Oracle 10g Server on Red Hat Enterprise Linux 5 (Deployment Recommendations)http://www.redhat.com/f/pdf/rhel/Oracle-10-g-recommendations-v1_2.pdf

Red Hat Enterprise Oracle Tuning Guidehttp://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Oracle_Tuning_Guide/RHELTuningandOptimizationforOracleV11.pdf

Red Hat Enterprise 5 IO Tuning Guidehttp://www.linux.com/index.php?option=com_rubberdoc&view=doc&id=12&format=raw

Red Hat Enterprise Linux Technology Capabilities and Limitshttp://www.redhat.com/rhel/compare/

Oracle Database

Oracle database feature matrixhttp://www.oracle.com/us/products/database/product-editions-066501.html

Oracle AMM vs Manual allocationhttp://www.dba-oracle.com/t_amm_automatic_vs_manual_ram_pools.htmhttp://www.dba-oracle.com/t_oracle_11g_new_feature_causes_amm_to_restart.htm

Oracle Performance Tuning Guide 11ghttp://download.oracle.com/docs/cd/B28359_01/server.111/b28274.pdf

Optimizing and Protecting Storage with Oracle Database 11g Release 2http://www.oracle.com/technetwork/database/focus-areas/storage/database-11gr2-managing-storage-whi-131523.pdf

18

Oracle ORIONhttp://www.oracle.com/technetwork/topics/index-089595.html

Oracle 11g Automatic Memory Managementhttp://www.datasoftech.com/library/DSI_11g_AMM.pdf

Storage

Back-of-the-Envelope Database Storage Designhttp://www.oracle.com/technetwork/database/back.pdf

Oracle IOPS and HBA Queue Depthhttp://www.ardentperf.com/2008/03/13/oracle-iops-and-hba-queue-depth/

Storage Array Capacity: Performance vs. Cost (SSD)http://www.techrepublic.com/blog/datacenter/storage-array-capacity-performance-vs-cost/2239

SAN performance best practices: More ways to avoid bottleneckshttp://searchstorage.techtarget.com/tip/0,289483,sid5_gci1527230,00.html

RAID 5 SSD I/O Performance Charthttp://www.tomshardware.com/reviews/enterprise-flash-ssd,1971-16.html

Short Stroking: How it Works (Increasing IOPS by reducing storage capacity)http://www.tomshardware.com/reviews/short-stroking-hdd,2157-2.html

Faster Oracle Performance with HyperStor Solid State Storagehttp://www.vion.com/Documents/HS%20Oracle%20Whitepaper%20rev%203.pdf

Understanding Storage’s Impact on Oracle Performancehttp://www.nyoug.org/Presentations/SIG/LI/Texas_memory_systems_storage_impact_on_db_performance.pdf

Leveraging EMC CLARiiON with Enterprise Flash Drives for Oracle Database Deploymentshttp://www.emc.com/collateral/hardware/white-papers/h5967-leveraging-clariion-cx4-oracle-deploy-wp.pdf

Standardized Storage Benchmarkshttp://www.storageperformance.org/home

Storage Performance Calculatorhttp://www.wmarow.com/strcalc/

Processor Performance

Standardized Processor Benchmarkshttp://www.spec.org/

Passmark CPU Benchmarkshttp://www.cpubenchmark.net/cpu_list.php

19