Pre-Con Ed: Simplify Administration and Reporting in CA Application Performance Management 10

World®’16

Pre-ConEd:SimplifyAdministrationandReportinginCAAPM10

GuenterGrossberger,Sr.EngineeringServicesArchitect,CATechnologiesAndreasReiss,Sr.PrincipalEngineeringServicesArchitect,CATechnologies

DO4X152L

DEVOPS– AGILEOPERATIONS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

CAAPMCommandCentermanagesyouragentconfigurationstosimplifytheadministrationofCAAPMinstallationsinevenlargescaleenvironments.YouwilllearnhowtocreateandupdateagentconfigurationsintheCommandCenterUIandpushthemtoagents.Wewillalsodiscusshowtomonitorcontainerizedapplications,howtousecreatereportsusingSQLwithCAAPMandhowtouseRESTbasedAPIstocorrelatereporting.

GuenterGrossberger

AndreasReissCATechnologies,EngineeringServicesArchitect

CATechnologies,Sr.EngineeringServicesArchitect


Agenda

APMCOMMANDCENTER:AGENTPACKAGES ANDBUNDLES

PACKAGEUPGRADESANDDYNAMICEXTENSIONS

MONITORINGMODERNAPPLICATIONS

REPORTINGWITHCAAPMSQL

REPORTINGWITHCAAPMRESTAPIS

1

2

3

4

5

Q&A6


CAAPMCommandCenter:AgentPackagesandBundles


Recap:CommandCenterFocusAreas

§ CommandCentercanbedistilledinto4distinctareas

Diagnostics Health

ViewInformationaboutJavaagentsrunningwithintheenvironment

Remotelyfetchagentlogfilesandotherdiagnosticinformation

Packagediagnosticsforuploadtosupportticket

RemotelychangetheLogLevelonasingleJavaAgent

Deployment

Abilitytocreatetailoredpackages fordeploymenttoproductionsystems

PackagesexposedviaAPIforconsumptionby3rd Partydeploymenttools

Cloudbasedstoreforagentrelatedpackages:CAAPMMarketplace

Configuration

Abilitytocreateconfigurationgroups

Create&enforceconfigurationpolicies

Supportfor‘hot’and‘cold’configuration

Audittrailofconfigurationchanges

Suggestadditionalmonitoringopportunities

Reviewthemonitoringoverheadoftheagents

Reportissues/errorswiththemonitoring

Suggestchangestothemonitoringtoreduceoverhead


Recap:Architecture


WhatisaPackage?

Apackageisanarchive (ziportar)thatcontainsallofthematerialsrequiredtodeployanAPMJavaAgent;thisincludes:

§ AgentJAR

§ Agentrofile andPBDs

§ OthersupportJARsandbinaries

§ Installinstructions


WhatisaPackage?

Apackageistailoredtotheenvironment beingmonitored

§ Applicationserver

§ Requiredmonitoringfunctionality

§ Specificmonitoringcapabilities

§ Specificinstallationrequirements

Apackageismadeupofoneormorebundles,eachofwhichrepresentsapieceoffunctionalityofferedbytheagent


HowIsaPackageBuilt?

Creatingapackageinvolvesusinga4stepwizardto:

1. Definetheenvironment

2. Tailorthebundles

3. Tailortheconfiguration

4. Tailortheinstallationinstructions

Skippingsteps2-4willcreateadefaultpackagefortheselectedenvironment


WhatDoesaBuiltPackageLookLike?

Apackageisaziportarfilethatexactlyresemblesanagentinstalldirectory,withtheusual‘wily’rootfolder

Additionalfilescreated:

§ installInstructions.md – providestailoredinstructionsonhowthispackageshouldbedeployed

§ manifest.txt – provideslistofbundlesused,togetherwithanypropertyoverrides

§ acc-master.pbl – usedtoprovideanexactlistofpbdfilesincludedintheconfiguration

wily

core

config

tools

acc-master.pbl

IntroscopeAgent.profile

pbdfiles

hotdeploy

ext

manifest.txt

installInstructions.md

Agent.jar

common

connectors

logs


CAAPMCommandCenter:CreateanAgentPackage


CreateNewAgentPackage

2.ClickNew

1.ClickPackages



2.SelectOS

1.Enternameandcomment

3.Selectagentversion

4.EnterMOM/EM(optionalorlist)

5.ClickDONE



Downloadlink

OOTBrequiredandrecommendedbundlesper

applicationserver

Version



Downloadlink

Installandconfigurationinstructions


Demo:CreateAgentPackage


CAAPMCommandCenter:CustomizeAgentPackage


CustomizeAgentPackage

1.Selectagentbundle

2.Click“DONE”

Or“COPY”



Bundledescriptionanddependencyinformation

Bundlesareorganizedintofourcategories:

1. Core2. Environment3. Feature4. Custom

1.Selectbundle

3.Click“ConfigureBundles”

2.Click“Add”



§ IntroscopeAgent.profile isbrokendownintobundles– EnterpriseManagerconnection– Javaagent– CommandCenter– Browseragent– JMX– EJB

§ CommandCenterconstructspackageIntroscopeAgent.profile frombundles



Changedpropertiesareshowninboldfont

Youcanadd,revertanddeleteyourownpropertieshere.

Descriptionandpropertiesareshown

perbundle

E.g.use“MathClient”asagentname

Enteracustomproperty

Custompropertiesaremarkedwiththestar



Youcanedittheinstallinstructionsaswell.

2.Click“DONE:whenyou’refinished

1.Click“InstallInstructions”

Textismarkdownsyntax(helpexplains

markdown!)


Demo:CustomizeAgentPackage


CAAPMMarketplace


Motivation

§ CAAPMisextensible

§ Therearemanyfieldextensionsoutthere– where?

§ CAAPMCommunity,github.com/CA-APM,colleagueslaptop,etc.

§ WehavestartedtomakeextensionsfullysupportedviaCASupport

§ CAAPMMarketplaceistheplacetofindallCAAPMextensions

§ http://marketplace.ca.com/apm


CAAPMMarketplacehttp://marketplace.ca.com/apm

CASupportedextension

Searchorselectcategory

Searchorselectcategory

• Agentextensions• EPAagentplugins/scripts• Examples

SSO:ca.com,Support,Communitieslogin

AllCAAPMextensionsarefree!


CAAPMMarketplace

§ Afterorderingyouwillgettwoemails– Orderconfirmation

– DownloadLink

§ Downloadedfile(ifagentextension)– <extension>-<version>-dist.tar.gz

e.g.spring-3.3.0-dist.tar.gz§ post-10.5/spring-3.3.0.tar.gz(agentbundle!)

§ pre-10.5/spring-3.3.0-pre-10.5.tar.gz(agentbundle!)

– ImportintoCommandCenter:Importbuttonin10.5,copytoimportfolderin10.2,10.3

http://marketplace.ca.com/apm


ImportExtensions

2.ClickImportandselectfile

1.ClickBundles


ImportExtensions

Restartnecessary

Bundlewillappearinlist


Demo:CAAPMMarketplace


Lab1:Create&CustomizeAgentPackage


DynamicExtensions


Motivation

§ Agentchangesoftenrequireapplicationserverrestarts

§ Applicationservermanagedbydifferentteam

§ Requireschangeboardapproval

§ PBD changesareautomatically reloaded

§ IntroscopeAgent.profile changedareautomatically reloaded

§ Whynotextensions?


DynamicExtensions

Newagentsub-directories

Todeployandagentextension:1. UseCommandCenteror2. Copyextensioninto$WILYHOME/deploy

Extensionwillbeextractedintoextensionsdirectory– Iftheextensionisdynamic(propertyinbundle.json)

§ Loadedversionofextensionisunloaded§ Newversionofextensionisloaded§ Noapplicationserverrestartrequired

– Otherwiseonlyloadedonapplicationserverrestart

Oldagentextensions:core/config,core/ext

DropnewextensionsintodeployoruseACC

Newextensionsdirectory


DynamicExtensions

Norestartnecessary

Bundlerequiresrestart

Dynamicextension

Old(pre-10.5)extension,not

dynamic


DynamicExtensions

Dynamicextensions

Bundlerequiresrestart

Dynamicextension


Youcanpushachangedconfigurationtoagroupofagents(“NamedFilter”)directlyfromtheCommandCenterUI

Agentstatuswillshowifanagentneedstoberestarted

DynamicExtensions


PackageVersioning

§ Eachpackagehasaversionassociatedwithit

§ Thisversionincrementsafter adownloadedpackageisedited

§ Thislimitsthenumberofversionsthatarecreated

§ Packageusagebyversionisalsoreported– Indicationthatagentisusingoutofdatepackage

Version=1

Version=1

Version=1

Zipv=1

Version= 2

Version=1

Download


TrackingPackageUsage

CommandCenterreportspackageusage

§ PackagedetailsareincludedinprofileandexposedasJavasystemproperties###################################

# PACKAGE DETAILS###################################

acc.package.id=1acc.package.name=tomcatLinux

acc.package.version=1

§ CommandCentercontrollerpicksupthesepropertiesandpassesontoconfigurationserver

§ Configurationserverreconcileswithpackageinordertoreportpackageusage


Demo:DynamicExtensionsandPackageUpgrades


MonitoringModernApplications


ApplicationsareArchitectedDifferentlyMicroservices

(Agile,loosely-coupled,independent,continuousdelivery)Monolithic

(Tightly-coupleddependent,waterfall,changeintolerant)


Microservices:MonitoringScale&ComplexityMicroservices architectureisanemergingtechniquewhichpromisestodeliverflexibilityandscalabilitytothedevelopmentanddeploymentofservicebasedapplications

http://www.pwc.com/us/en/technology-forecast/2014/cloud-computing/features/assets/feature02-figure02.jpg


MonitoringMustScale withNewArchitecture

MicroservicesArchitecturewitheachelementoffunctionalityinaseparateservice….

Scalesbydistributingservicesacrossservers,replicatingasneeded

MICROSERVICE CONSIDERATIONS

§ Testingacrossmultipleservices§ Releaseco-ordination§ Manymovingparts/multipleinstances§ Manynewdatastores– SQL,NoSQL§ Networklatency,messaging,emergent

behaviors§ Complexity!


Containers– MonitoringComplexity&ScaleContainersincludetheapplicationandallofitsdependencies,butsharethekernelwithothercontainers;Dockercontainersarelightweight,runonanycomputer,onanyinfrastructure,inanycloudandaregreatformicroservices


Cloud– MonitoringScaleCloudcomputingisaboutaccesstoapoolofcomputingresources(servers,storage,networks,servicesetc)thatcanberapidlyprovisionedandmadeavailablewithminimaloverhead–NIST


Browser

Mobile

Network WebServer

SyntheticTransactions

SQLDatabase

APIGatewayAppServer

PassiveEUE(Agentless)

Mainframe

TransactionPerformance

=MonitoringAgent

Microservice

DockerContainer

SSO UserDataStore

Abilitytotracecomplextransactionsthattraversetheentireinfrastructurefromcustomer-facingsystemsofengagementtoback-endsystemsofrecord

SYSTEMSOFENGAGEMENT

SYSTEMSOFRECORD


E.P.I.C.A

PM

Smartagent• Collectwhatyouneedwhenyouneedit

Monitoringatscale• Monitor@scaleandclustering

Easeofadministration• GetcontrolwithCAAPMACC

Simplyvisualizelargecomplexdata• CAAPMHomePageandAssistedTriagehelpquickly

identifyanddiagnoseproblemsthatmatter

Elegance@Scale


APMSupportforModernTechnologies§ MonitorappsdeployedtoDockercontainerswithAPMagent§ Collectsbasiccontainerhealthmetrics(fieldpack)§ VisualizeappperformanceandDockerhealthinoneview§ https://github.com/CA-APM/DockerMonitoring

§ EasilymonitorappsdeployedusingCloudFoundryjavabuildpack§ CollectsbasichealthmetricsfromPivotalCloudFoundryloudFoundry(fieldpack)§ VisibilityintoapplicationperformanceandCFPivotalhealth§ MembersoftheCloudFoundryFoundation

§ CAAPMagentsarefullysupportedandprovidetransactioncorrelation,deepdiverootcauseanalysis;browseragentforenduseranalytics

§ FieldpacktoreportCloudwatch dataintoCAAPM

§ CAAPMpluginintegratesforbuildperformancecomparisons&attributeupdates§ https://wiki.jenkins-ci.org/display/JENKINS/CA+APM+Plugin

§ Node.jsagentsarefullysupportedandprovidetransactioncorrelation,deepdiverootcauseanalysis;browseragentforenduseranalytics

§ https://www.npmjs.com/package/ca-apm-probe


ContinuousIntegration/Deployment:MonitoringforResiliencyandChangeContinuousintegrationisaprocessinwhichalldevelopmentworkisintegratedasearlyaspossibleandtheresultingartifactsareautomaticallycreatedandtested;thisprocessshouldidentifyerrorsasveryearlyintheprocess

Source:http://cdn.rancher.com/wp-content/uploads/2015/11/18175501/ci_flow.png


MonitoringModernApplications

§ JavaandNode.jsagentinDockerisfullysupported

§ JavaagentincludedinCloudFoundry JavaBuildpack

§ ExtensionsontheCAAPMMarketplace(http://marketplace.ca.com/apm):– Docker– AWS– MongoDB– Jenkins– CloudFoundry


ReportingwithCAAPMSQL


E M

Commandlineworkstation

IntroscopeJDBCdriver

Webservices

CAAPMdatafeedTheProblem


ASolution!


MapMetricDatatoaSingleSQLTableColumn_Name Column_Type

attribute_type Long

source_name String

domain_name String

agent_host String

agent_process String

agent_name String

metric_path String

metric_attribute String

frequency Long

ts timestamp

agg_value Long

max_value Long

min_value Long

value_count Long

agent

metric

values


MapMetricDatatoaSingleSQLTableColumn_Name Column_Type

attribute_type Long

source_name String

domain_name String

agent_host String


agent_name String

metric_path String


frequency Long

ts timestamp

agg_value Long

max_value Long

min_value Long

value_count Long

agent

metric

values

§ Onetable:numerical_metric_data

§ Attribute_type:determineshowagg_value,max_value… areused

§ Metric_attribute:“ResponsesPerInterval”,“AverageResponseTime(ms)”…

§ ts andfrequencymustbepresentineveryquery


ExampleUseCase:GetSlowestApplications

DianawantstoaskCAAPMSQL

whichapplicationsareherworstperformersduringthelastweek

sothatshecanknowwhichneedthemostattentionandwhichareimproving


ExampleUseCase:GetSlowestApplications

SELECT agent_host,agent_name, metric_path,

Sum ( value_count ) AS total_count, Avg ( agg_value ) ASmean_agg

FROM numerical_metric_data

WHERE metric_path LIKE ’Frontends|Apps|%'

ANDmetric_attribute LIKE 'Average Response Time (ms)'

AND ts BETWEEN {ts '2016-11-0612:00:00.0'} AND {ts '2016-11-1412:00:00.0'}

AND frequency = 3600

GROUP BY agent_host, agent_name, metric_path

ORDER BYmean_agg DESC,total_count DESC, agent_host, agent_name, metric_path


Demo:CAAPMSQLColumn_Name Column_Type

attribute_type Long

source_name String

domain_name String

agent_host String


agent_name String

metric_path String


frequency Long

ts timestamp

agg_value Long

max_value Long

min_value Long

value_count Long


ReportingwithCAAPMRESTAPIs


RelationshipView

Alertstatus

Spikegraphs

Basicandcustomattributes

Filters

TimelinechangeeEvents

Perspectives

DA

Alert


APMTeamCenterVocabulary

Vertex

Edge

VertexGroup


CAAPMTeamCenterVocabulary– Attributes

§ Whatisanattribute:ametadatathatisassociatedwithavertexanddescribesavertex

§ EachvertexcomeswithOOTBattributes– agent– applicationName– hostname– name– type


CAAPMTeamCenterAttributes,Continued

§ TeamCenterhaspredefinedspecialattributes:– Location– Owner– Region– Tier

§ Candefineandassociatecustomattributeswitheachvertex


AttributeBenefits

§ Createmultiplelayersbygrouping

§ EndUserfriendlynames

§ Groupelementswithidenticalcontextintocommongroups

§ Assignaccess/permissions

§ How?By:– Groupingvertexes– Renamingcomponents(throughgrouping)– Annotatingvertexes– Highlightingvertexes


CAAPMTeamCenterAPI

§ CAAPMRESTAPIallowsyoutocreate/update/deletevertexattributes– Listsexistingvertices– Listsexistingattributesandtheirvalues– CreatesnewCUSTOMattributeforexistingvertex– Updatesexistingattributeforexistingvertex

§ Canbeusedinscriptsforautomation

https://docops.ca.com/ca-apm/10-3/en/integrating/api-reference-guide/apm-rest-api


CAAPMTeamCenterAPI inAction– GetVertices


GetVertexIDAttributesList


UpdateVertexwithNewAttributeValue


ChecktoSeeifOwneris“Andrew”


Demo:CAAPMTeamCenterRESTAPI


APMTeamCenterRESTAPI• Queryverticesand

attributes• Createorupdateattributes• Useinscripts

CAAPMCommandCenter• Createagentpackages• Selectandcustomize

bundles• CAAPMMarketplace• dynamicextensions

CAAPMSQL

• CAAPMfromanyUI• Onesingletable• Fast,powerfulreports

SummaryAFewWordstoReview


Thankyou.

Stayconnectedatcommunities.ca.com

@CAWORLD#CAWORLD ©2016CA.AllRIGHTSRESERVED.75 @CAWORLD#CAWORLD

DevOps– AgileOps

FormoreinformationonDevOps– AgileOps,pleasevisit:http://cainc.to/wYXSg6

Technology

Pre-Con Ed: Simplify Administration and Reporting in CA Application Performance Management 10