Michael Zechman Database Consultant, Convertible Technology, Inc

Automating Upgrades

Presented to:zSeries Oracle Special Interest Group ConferenceRedwood Shores, CaliforniaMarch 28 – April 2, 2004

Michael ZechmanMichael ZechmanDatabase Consultant, Convertible Technology, Inc. Database Consultant, Convertible Technology, Inc.

Defense Enterprise Computing Center (DECC) Defense Enterprise Computing Center (DECC) Mechanicsburg, PennsylvaniaMechanicsburg, Pennsylvania

2Our People Make a Difference Every Day.

Presented at:zSeries Oracle SIG Conference 2004 Our People Make a Difference Every Day.

Agenda

• How can Automating Upgrades benefit zSeries Oracle Customers

• Why Automate

• What is required for an Upgrade

• Upgrade Automation

• Summary

• Questions and Automation Discussion



How can Automating Upgrades benefit zSeries Oracle Customers

• Why should I be Listening to this on April Fools Day! Mike’s Top 10 Reasons:

– Recommended to Jim an Automation Session Per SIG. – Share automation techniques among Oracle SIG’s very unique

zSeries user community– Upgrades now take less than 5 person-minutes per Database– Your site might not require automated Upgrades, but

• Pieces of it may be useful• Might spawn ideas for your future automation• Information sharing on Automation (other than Upgrades) will be

discussed– Upgrade discussion was picked for a reason – It was easy to

pull pieces, easy to create, and use.



How can Automating Upgrades benefit zSeries Oracle Customers

• It is the Second session in the afternoon?

– If you are overloaded with work, consider automating processes (or your manual procedures)

– Do your customers demand instant attention? Think about adding dynamic automated services for those types of requests

– Generalized Method of how to discuss and share automation– Don’t miss putting Jim on the spot! – Prize Time – A picture of something I bought last Year! I have

been waiting to get zSeries input. Only a zSeries Group may be able to help me convince my wife to appreciate it. Your should hear the initial and monthly comments from her on it! I need help! Best comment wins a Prize!



Why Automate

• Mainframe Computing– 7 “Hosts or Boxes”; one split– 45 Logical Partitions (LPARs)– 4,713 Millions of Instructions Per Second (MIPS)

• Mid Tier– 230 servers hosted– Windows, Unix, LINUX Operating Systems

• Storage– RAID DASD

• EMC, Spectris, IBM RAMAC, HDS 7700E, IBM Enterprise System Server• 25,544 GB of DASD

– Tape storage• 308,000 tapes in inventory, 1097 tapes processed on an average day• 27 Tape Silos, 3 Virtual Tape System (VTS), 1 Automatic Transport Unit (ATL) for mid-tier, 108

stand-alone tape transports

• 425 Applications and 7 Different types of Databases (Software)



Why Automate

• Our Team - Oracle and IDMS

– 160 + Database Systems to Manage

• 70 Production• 45 Oracle zSeries databases, 15 production

– Mainframe, UNIX, and Windows platforms– Service organization providing computing

services to our defense customers– Protect Integrity of Database– Staff of Nine– Team generated Automation and Support!



Why Automate

• More Workload and Less Staff

312189186

546321

235

598512

257

853

656

309

964722

322

1151976

322

1390

1183

337

2386

1889

323

3085

1951

338

3410

2200

324

4576

2554

319

4713

2554.4

315

4950

2675

287

0500

1000

1500

2000

2500

30003500

4000

4500

5000

'93 '94 '95 '96 '97 '98 '99 '00 '01 '02 '03 Current TransGoal

CPU MIPS

DASD GB

Staffing

(DASD * 10)December 2003



Why Automate

• Customer test environments that are demanding - Aside from normal scheduled backups etc., complex application testing environments require dynamic database service that we must support:

– Backup– Restore– Restore PIT (planned, but not automated yet-done manually)– Clone/Copy of a database – Cross LPAR Support For Clone/Copy– Startup– Shutdown– Recycling– Database Changes– Special Case Backups– Cross LPAR Support For Special Backup Saves– Database Software Upgrades– Of course, at 4:00 PM Friday or during evening Hours!

• During the Last Fiscal Year – Not including scheduled jobs, dynamic automation supported 1550+ Dynamic Requests From our customer application DBA’s without our involvement! Most features not turned on until late summer 2003.



What is Required For an Upgrade

– Unload Software or Downloaded Cumulative Oracle Patches– Prepare Load Libraries For Use (receive and authorize etc.)– Database Upgrade Capable – Must validate and change prior to

upgrade– Possible Changes to Current Automated Scheduled Processes– Backup Before and After– Startup and Shutdown of Database– Startup and Shutdown of Services (Instance and Listener)– OSDI Parameter Changes– INITORA Changes (multiple times)




– Software Library Alias Names Changes– Oracle .SQL Executions and Validation (CATPATCH etc.)– Unique Changes (For example remove RBS’s add UNDO or

change all to locally managed)– Validation of Database Entities and successful Upgrade– Security Updates To Database Once Upgrade Complete – Validation Database Passes Security Requirements– Document Date, Time, Technician Performing Upgrade– Archival of Upgrade Output– Database Validation Upon Successful Upgrade




Upgrade issues we had: – Multiple upgrades have to be done to a database per year– Time consuming – Repetitive work– Error prone especially when doing multiple upgrades at a time – Takes 2 to 4 person hours for each database upgrade after

libraries are prepared– We had to perform more upgrades than the number of

databases we have. The overall number is based on the number of baseline application backups/saves used for application testing



Upgrade Automation

• Our Requirements

– Get out of Dynamic work requests. It was killing our productivity!

– Application DBA’s must NOT be able to perform functions with direct access. This would be a security violation.

– Protect integrity of overall database system– Must be bullet proof – upgrade validation was a priority– Must be able to track history of changes and requests– Provide capability for customers to upgrade test databases– Provide capability for customer to upgrade various database

backups – Many different baseline testing backups/saves– Possibly provide capability for customer DBA’s to upgrade

production



Upgrade Automation

Upgrade Automation System

–Took 4.5 person-weeks to create–Used pieces we had, pulled from web, and

created ourselves– ISPF Panels for online validation and

request submission–15+ ISPF Edit Macros – Batch mode–10+ REXX Processes–15+ CLIST to execute ISPF EDIT Macros



Upgrade Automation

Upgrade Automation System

–3 Assembler Modules–215 Job Steps not including backups–Scheduling Product• Control-M Schedules• Control-O Rules

–Messaging used to communicate between processes–Batch ISPF, very nice tool–Batch SDSF, cool tool



Upgrade Automation

First Phase - Not in our Automation

• 1 Step per LPAR or site• Via Oracle’s ISPF Panels, manual, or via

Oracle’s new CD installation (specifics unknown at time of presentation creation. Note: Being presented at SIG by Oracle)

• Unload software or downloaded cumulative Oracle patches

• Prepare load libraries for use (receive and authorize etc.)



Upgrade Automation



Upgrade Automation



Upgrade Automation



Upgrade Automation



Upgrade Automation

Job name, and job number are in

position 81-100



Upgrade Automation

– Is Database Upgrade Capable? Validation and changes prior to Upgrade• Oracle Upgrade Validations SQL’s Used • Minor modification for automated validation purposes• A ROWCHECK Procedure was used for Validation• Upgrade Only Continues or Changes are made based on Return Codes

– Possible Changes to Current Automated Scheduled Processes• Batch ISPF Edit Macros Used• Inserts IEFBR14 and // to stop automated procedures/schedules

– Backup Before and After• Messages Sent ‘FORCE JOB …’

– Startup and Shutdown of Database• Normal Startup and Shutdown Batch Streams. • Parameters added/changed based on Exclusive or Migrate mode etc



Upgrade Automation

–Startup and Shutdown of Services (Instance and Listener)• Dynamic Starting and Stopping of Services• Job step was set to ‘Wait’ until service is

Stopped/Started• Used REXXWAIT Assembler module from Web• Created Customized REXX SERVSTAT to call

REXXWAIT• SERVSTAT - Customized REXX to perform

Operator Commands – Loops and waits



Upgrade Automation

REXXWAIT and SERVSTAT execution messages



Upgrade Automation

–OSDI Parameter Changes• If needed, ISPF macros and ALTER SERVICE

commands

– INITORA Changes (multiple times)• Batch ISPF EDIT Macros• IEBGENER• JOBPARM module to pass parameters



Upgrade Automation

• Sample ISPF EDIT macro:

VPUT in CLIST and VGET in Macro can be used to pass data to and from EDIT

Macro



Upgrade Automation

• ISPF Batch Sample Execution:



Upgrade Automation

JOBPARM Sample - Passes variables to CLIST/EDIT Macro:



Upgrade Automation

– Software Library Alias Names Changes• Dynamic Syntax Creation for IDCAMs• Batch ISPF Edit Macro’s• IDCAMS

– Oracle .SQL Executions and Validation • SQLPLUS• ROWCHECK• WHENEVER EXIT xxxx

– Unique Changes (For example Remove RBS’s add UNDO or Change all to locally managed)• Batch SQLPLUS• Batch ISPF Edit Macros• REXX code (if required)

– Validation of Database Entities and successful Upgrade• SQLPLUS• ROWCHECK

– Security Updates To Database Once Upgrade Complete• Security Scripts Execute



Upgrade Automation

–Validation Database Passes Security Requirements• Done during testing, but we have an automated

JCL Procedures to perform this task

–Document Date, Time, Technician Performing Upgrade (Done, not implemented)

• ISPF EDIT Macro• IEBGENER



Upgrade Automation

– Archival of Upgrade Output SPOOL TOOL – Database team developed. Takes SDSF Spooled Jobs or Step DD Output and Writes it to an Archive File

Useful Tool!



Upgrade Automation

–Database Validation Upon Successful Upgrade• SQLPLUS• WHENEVER EXIT xxxx• ROWCHECK



Upgrade Automation

• How do you permit non authorized DBA’s to perform these functions while still protecting the integrity of the system?

• Under USER Authority– Create a short lived dataset based on name of instance and function desired. User must have authority

to create it. The existence of this dataset is checked later to validate authority.– Send Message (causes message to be written to SYSLOG)

• Under SYSTEM Authority– Control O Detects Message– CTOSERV Deciphers Messages and Passes Parameters to REXX EXEC– REXX Exec Validates Parameters and executes unique REXX code for each Function– Code verifies that the short lived dataset created above exists. If it doesn’t, then user is trying to

manually send a message. ‘Gotcha’– A Batch Job is then submitted under our group’s scheduling user-id which is monitored by Control-M– Messages are sent to user when various phases are complete– Summary file is updated – Since Control-M monitors the job, any failures are quickly detected and resolved.



Upgrade Automation

• The picture I discussed earlier

– If you were a woman that did not need computer equipment, what would you not want your husband to bring home?



Summary

– Secured– Time consuming and redundant effort – Each

upgrade manually took 2 to 4 person-hours after libraries were prepared• Automated < 1-5 minutes technician time• Saved time already

– Dynamic requests while being controlled and monitored by operations via Control-M/O (standard scheduling/monitoring facility)

– History of requests and job executions



Summary

–Easy to add on for future releases. –Estimated 4-6 hours to update process for

each new release.• JCL PROC Change• Panel Change• ISPF EDIT Macro Change

–Messages sent to user after each phase.–Customers happy–Permits customer/our team flexibility–More time to take on new database tasks



Summary

– We are happy– Not perfect upgrade system

• Must be customized for each release• Occasional restarts required (testing phase)• Restarts not easy due to return code ‘IF’ etc. in PROC



Summary

• Future Automation–SPOOL TOOL integration–Restore point in time recovery –Clone/Copy Hot Backup to a test database–Clone/Copy Databases Between Data

Centers?–Use on our UNIX systems? Conversion

seems inevitable. Some tools exist. Suggestions?



Questions and Automation Discussion

–Questions or comments

–Automated Discussion• Share Site Automation on SIG?•Other Automation

–Thank you and have a great conference

Documents

Michael Zechman Database Consultant, Convertible Technology, Inc