Case Study: Verizon - CA Workload Automation ESP Edition: Best Practice’s, DevOps, Change Management Utilizing Endevor

World®’16

CaseStudy:Verizon- CAWorkloadAutomationESPEdition:BestPractice’s,DevOps,ChangeManagementUtilizingCAEndevor®DaveKovaleski - MTSIVSpecialist- Verizon

MFX138S

MAINFRAMEANDWORKLOADAUTOMATION

2 @CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation

3 @CAWORLD#CAWORLD

Abstract

VerizonTelecommunicationsmadeCAWorkloadAutomationESPEdition(CAWAESP)thestandardtoolforworkloadmanagementacrosstheWirelineandWirelessorganizationforbothmainframeanddistributedsystemsusingtheCAWAESPagenttechnology.Inthissession,theywillshareinsightsonhowtheysuccessfullyestablishedCAWAESPpresencebyinstillingVerizon'sschedulingbestpractices,DevOps,andchangemanagementutilizingCAEndevor®.

DaveKovaleskiVerizonMTSIVSpecialist

4 @CAWORLD#CAWORLD

VerizonBestPractices,DEV-OPS,ChangeManagementUtilizingCAEndevor®DaveKovaleski

November16,2016

5 @CAWORLD#CAWORLD

Agenda

INTRODUCTION

SCHEDULINGBESTPRACTICES

Q&A

DEV-OPSPROCESSING

CAWAESPANDCAENDEVOR®

SUMMARYANDCLOSINGREMARKS

1

2

3

4

5

6

6 @CAWORLD#CAWORLD

Introduction

VerizonITusesbestpracticesthatsupportandenableaworldclasscustomerexperience.TheProductionControl

departmenthasimplementedbestpracticeswhichincludeusingCAWAESPglobalvariablesandsymlibsasacentral

repositorytostoreapplicationartifactssuchasagent,path,anduserid’s.ThelargeapplicationsprocessedbyVerizon

benefitfromthevaluesthatareusedgloballyacrossapplications.Otherbestpracticesincludeimplementingnaming

standardsrelatedtoeachapplicationt hatsupportDEV-OPSsolutionsforapplicationteams.SomeofVerizon'sapplication

teamsperformtheirownapplicationstopandstarts.Inaddition,VerizondevelopersarenoweffectivelyusingCAWAESP

WebServicesinaDEV-OPStool.VerizonalsohasintegratedtheCAEndevor®SoftwareChangeManager(CAEndevor®)

softwaretotrackapplicationchangesthroughtheCAEndevor®software.

7 @CAWORLD#CAWORLD

ExperiencesVerypositivefeedbackfromtheapplicationteamsandschedulers.CommentsInclude:“Provideonestopupdateformanagingservernamechanges”&“Easytomigratefromtesttoprodbyupdatingvariablesinonelocation!”

BestPracticeDefiningandCreatingGlobalVariablesorSymboliclibrariestostoreagent,scriptname,UserIDinformation.

FindingsTheuseofGlobalVariablesandSymbolicLibrariesprovidesauserwithacentralizedlocationtoupdateandreferencevaluesthatwillbeutilizedgloballythroughoutanapplication.

BestPracticesGlobalVariablesandSymbolicLibraries

8 @CAWORLD#CAWORLD

BestPractices

Mainframe GlobalVariableandSymlib(Member) Variable Name=‘Value’

IPSNESP.CMD.CNTL.SYMLIB.VISN(CMDNS) AGENT='TWASTARDP'FILE_PATH='/starload/DNS'LOG_PATH='/starload/DNS/logs'SCRIPT_PATH='/starload/DNS/process'USERID='5staradm'

GlobalVariableandSymbolicLibraryExample

9 @CAWORLD#CAWORLD

BestPractices

CAWAESPApplicationandJobDefinition EventSimulation

APPLCMD#DBCU

INVOKE'IPSNESP.CMD.CNTL.SYMLIB.VISN(CMDNS)‘

UNIX_JOBDNSCM_BATCH_CBR_UPDATERUNDAILYAGENT%AGENTARGS%SCRIPT_PATH./set_env_dns_callmanager.ksh>>+

%FILE_PATH./callmanager/batchCbr.log2>&1USER%USERIDSCRIPTNAME%SCRIPT_PATH./dnscm_batch_cbr_update.ksh

ENDJOB

INVOKEIPSNESP.CMD.PROCLIB.VISN(CMD#DBCU)APPLCMD#DBCU

INVOKE'IPSNESP.CMD.CNTL.SYMLIB.VISN(CMDNS)'AGENT='TWASTARDP'FILE_PATH='/starload/DNS'LOG_PATH='/starload/DNS/logs'SCRIPT_PATH='/starload/DNS/process'USERID='5staradm'

UNIX_JOBDNSCM_BATCH_CBR_UPDATERUNDAILYAGENTTWASTARDPARGS/starload/DNS/process/set_env_dns_callmanager.ksh>>

/starload/DNS/callmanager/batchCbr.log2>&1USER5staradmSCRIPTNAME/starload/DNS/process/dnscm_batch_cbr_update.kshENDJOB

GlobalVariableandSymbolicLibraryExample

10 @CAWORLD#CAWORLD

ExperiencesVerypositivefeedbackfromtheapplicationteamsandschedulers.CommentsInclude:“Ineedtofocusonmyworkloadandthesetupavoiddistractionsfromoutsideapplications!”

BestPracticeNamingstandardsaredefinedforapplications,eventsandprocedurelibraries.

FindingsTheuseofstandardizednamesallowsouruserstoonlyviewandaccesstheirworkloadwhensubscribingtoworkloadintheCAWAESPWorkstation.Thestandardsalsoalloweaseofmonitoringandsupportforbothouroperationsandapplicationteams.

BestPracticesApplication/Event/ProcLibraryNamingStandards

11 @CAWORLD#CAWORLD

BestPractices

EventNamingStandard

EventnamesaredefinedasEventprefix.Event name.Theeventprefixnamewillbedeterminedbywhatenvironmenttheapplicationisrunningon.

TheEventprefixisdefinedwiththefollowingformat:

$VZWXAAA

Where:

$VZWX-Thefirst4charactersoftheeventprefixwillalwaysstartwith$VZWXAAA- Thelast3characters(5-7)aredefinedasthe3alphaApplicationCharacterCode

TheEventnamewillbetheapplicationnamethatistriggeredbytheevent.

Example:$VZWXCMD.CMD#DBCU

| |Event Prefix ApplicationName

Application/Event/Proclib StandardizedNamesExample

12 @CAWORLD#CAWORLD

BestPractices

Application NamingStandard

Applicationnamesarelimitedto8charactersandaredefinedwiththefollowingformat:

AAAEXXXX

Where:

AAA- Charactersaredefinedasthe3 alph charactercodeE- CharactersaredefinedastheEnvironmentNameXXXX - CharactersaredefinedasFreeForm

Example:

CMD#DBCU

CMD# DBCU|| |

ApplicationEnvironment FreeForm


13 @CAWORLD#CAWORLD

BestPractices

MainframeProclib NamingStandard

MainframeProclibs containApplicationnamesarelimitedto8charactersandaredefinedwiththefollowingformat:

IPSNESP.AAA.PROCLIB.VISL

Where:

IPSNESP– DoesnotchangeAAA- Charactersaredefinedasthe3 alphacharactercodePROCLIB– DoesnotchangeVIS– DoesnotchangeL–MainframeLocationCode

Example:

IPSNESP.CMD.PROCLIB.VISN| |

Application North


14 @CAWORLD#CAWORLD

ExperiencesVerypositivefeedbackfromtheapplicationteamsandschedulers.CommentsInclude:“Controllingourworkloadreducesdowntimethatourcustomersmayhaveincurred!”

Dev-OpsOptionAnoptionwasimplementedbytheSchedulingteamtoallowapplicationteamstostopandstartalloftheirproductionjobs.Theschedulingtechniqueisheavilyusedwhenthereisscheduledmaintenance.

FindingsByimplementinganoptiontostopandstarttheirjobs,theapplicationteamcancontrolandnotrelyonoperationstostopandstarttheirworkload.ThisfacilitatesaDEV-OPSsolutiontoourapplicationteams

Dev-OpsApplicationTeamDev-OpsOptionStop/StartProductionJobs

15 @CAWORLD#CAWORLD

Dev-Ops

Step1:Create ThresholdResource

RESDEFOTA_PRD_DATA_OTA_EASTBATCHADDENTERPRISETHRESHOLD MAX(1)

StopandStartSetup

16 @CAWORLD#CAWORLD

Dev-Ops

Step2:Create StopFileTriggerApplication

APPL OTAECFT/*Thisappl containsallOTAEASTCFTjobs*/

FILE_TRIGGEROTAE_STOP_FT/*TriggersOTAEASTStopEvent$VZWXOTA.OTAESTOPRUNDAILY

AGENTOTAEASTFILENAME'/ota/data/adm_ota/otaStopEastESP'CREATENOCHANGE(1)CONTINUOUS($VZWXOTA.OTAESTOP)DUEOUTEXEC23:59EASTERNNOTIFYOVERDUEALERT(GLB5)

StopandStartSetup

17 @CAWORLD#CAWORLD

Dev-Ops

Step3:CreateStop Application

APPLOTAESTOP/*Thisappl willsetOTA_EASTBATCHto0*/

JOBSTOP.EASTOTA_BATCHTASKSELFCOMPLETING/*ThislinkjobwillstoptheOTAEastbatchjobs.*/RUNDAILYRELEASEADD(Cleanup_Stop_Start_Files.OTAEAST_STOP)ESPRESDEFOTA_PRD_DATA_OTA_EASTBATCHSETAVAIL(0)NOTIFYPNODE(COMPLETE)ALERT(SHTR)

ENDJOB

UNIX_JOBCleanup_Stop_Start_Files.OTAEAST_STOP/*ThisscriptwillcleanuptheESPStop/Startdummyfilescreatedtostop/startOTAbatchRUNDAILYAGENTOTAEASTUSERadm_otaSCRIPTNAME/ota/data/adm_ota/cleanupESP.sh

ENDJOB

StopandStartSetup

18 @CAWORLD#CAWORLD

Dev-Ops

Step4:Create StartContinuousFileTriggerApplication

APPL OTAECFT/*Thisappl containsallOTAEASTCFTjobs*/

FILE_TRIGGEROTAE_STRT_FT/*TriggersOTAEASTStartEvent$VZWXOTA.OTAESTRTRUNDAILY

AGENTOTAEASTFILENAME'/ota/data/adm_ota/otaStartEastESP'CREATENOCHANGE(1)CONTINUOUS($VZWXOTA.OTAESTRT)DUEOUTEXEC23:59EASTERNNOTIFYOVERDUEALERT(GLB5)

ENDJOB

StopandStartSetup

19 @CAWORLD#CAWORLD

Dev-Ops

Step5:CreateStartApplication

APPLOTAESTRT/*Thisappl willsettheOTA_EASTBATCHto1whenholdisreleased

JOBSTART.EASTOTA_BATCHTASKSELFCOMPLETING/*ThisjobswillstartallOTAEastbatchjobs.*/RUNDAILYRELEASEADD(Cleanup_Stop_Start_Files.OTAEAST_START)ESPRESDEFOTA_PRD_DATA_OTA_EASTBATCHSETAVAIL(1)NOTIFYPNODE(COMPLETE)ALERT(SHTR)

ENDJOB

UNIX_JOBCleanup_Stop_Start_Files.OTAEAST_START/*Thisscriptwillcleanupthe ESPStop/Startdummyfilescreatedtostop/startOTAbatchRUNDAILYAGENTOTAEASTUSERadm_otaSCRIPTNAME/ota/data/adm_ota/cleanupESP.sh

ENDJOB

StopandStartSetup

20 @CAWORLD#CAWORLD

ExperiencesAverypositiveexperiencethroughoutourapplication,schedulingandauditingteams.CommentsInclude:“Versioncontroliscriticaltobusinessresiliencyaswellassatisfyingauditrequirements.”

CAEndevor®WithCAWAESPCAEndevor®hasbeenintegratedwithCAWAESPtotrackapplicationupdatesandmanagejobchanges.

FindingsByimplementingCAEndevor®,thesolutionallowsustotrackallapplicationandjobchangesforauditingpurposes.Thetechniquealsoallowsaquickwaytoback-outchangesmadetoapplicationsthatmaybecausingproductionissues.

CAWorkloadAutomationESPEditionandCAEndevor®HowCAEndevor®isIntegratedWithCAWorkloadAutomationESPEdition(CAWAESPEdition)

21 @CAWORLD#CAWORLD

CAWAESPEditionandCAEndevor®

Step1:Creation oftheCAEndevor®Environment

The CAEndevor®AdministratorincoordinationwithourSchedulingteamdefinedtheCAEndevor®environment.TheCAEndevor®environmentcontainsthefollowing:1Environment,1System,27Subsystems,2stagesand2types(APL)forapplicationand(SYM)forGlobal Variables/Symlibmembers.

CAEndevor®Setup

EnvironmentEPSWLS

SystemEPS

SubsystemAAA00(AAA =3Character

Applicationcode)

StageStage 1:PSTGStage 2:PROD

TypeType1:APLType2:SYM

Step2:Creation ofthePackageShipmentRules

Sincethe CAEndevor®softwareisrequiredtobeinstalledonourdevelopmentmainframe,weneededaprocesstosendtheupdatedapplicationstotheproductionmainframeproclibs.TheCAEndevor®AdministratorcreatedPackageShipmentrulestoallowthistooccur.OncetheCAEndevor®packageisexecuted,theCAEndevor®withCAWAESPelementsarethen‘shipped’toourproductionproclibs.

22 @CAWORLD#CAWORLD


Step1:CAEndevor® ADDtoStage1

CAEndevor®WithCAWAESPEditionProcess

23 @CAWORLD#CAWORLD


Step1:CAEndevor®ADDtoStage1


24 @CAWORLD#CAWORLD




25 @CAWORLD#CAWORLD


CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess

26 @CAWORLD#CAWORLD

Step2:Package CreationandShipment


27 @CAWORLD#CAWORLD



28 @CAWORLD#CAWORLD



29 @CAWORLD#CAWORLD



30 @CAWORLD#CAWORLD



31 @CAWORLD#CAWORLD



32 @CAWORLD#CAWORLD



33 @CAWORLD#CAWORLD



34 @CAWORLD#CAWORLD



35 @CAWORLD#CAWORLD



36 @CAWORLD#CAWORLD



37 @CAWORLD#CAWORLD



38 @CAWORLD#CAWORLD



39 @CAWORLD#CAWORLD



40 @CAWORLD#CAWORLD


Step2:Package CreationandShipmentReportEmail


41 @CAWORLD#CAWORLD

JobChangeVerification


42 @CAWORLD#CAWORLD

JobChangeVerification


43 @CAWORLD#CAWORLD


Job ChangeVerificationandEventSimulation


44 @CAWORLD#CAWORLD

Summary

TheVerizonProductionControldepartmenthasimplementedamyriadofoptionsandscheduling

techniquestoenhanceourapplicationusersexperiencewiththeCAWAESPSchedulingtool.Theoptions

thatwereimplementedareBestPracticeswiththeuseofGlobalVariablesorSymlibs andNaming

StandardsaswellasDEV-OPSforenduserstostopandstarttheproductionjobsduringmaintenance

windowsandintegratingCAEndevor®withtheschedulerforversioncontrolmanagement.

VerizoncontinuestorolloutnewBestPracticesintheDEV-OPSarenaplusSelfServiceoptionsthatwill

permitapplicationteamstoholdorreleasebatchjobsandtheuseofCAEndevor®forapplicationteams

totracktheirlowerenvironmentchangeswithinCAWAESP.

45 @CAWORLD#CAWORLD

Questions?

46 @CAWORLD#CAWORLD

Thankyou.

Stayconnectedatcommunities.ca.com

47 @CAWORLD#CAWORLD

MainframeandWorkloadAutomation

FormoreinformationonMainframeandWorkloadAutomation,pleasevisit:http://cainc.to/9GQ2JI

Technology

Case Study: Verizon - CA Workload Automation ESP Edition: Best Practice’s, DevOps, Change Management Utilizing Endevor