Upload
percival-logan
View
255
Download
0
Tags:
Embed Size (px)
Citation preview
Job Scheduling: History and Evolution
November 17, 2010Welkin Associates
Job Scheduler Terminology and Components
The Scheduler Study Overview
Brief History of Job Scheduling
Job Scheduler Functions
Job Scheduler Architectures
Evaluation of Job Schedulers and Vendors
Q/A
Agenda
Terminology Job: An executable unit. Can be a systems command, a script, an application program, or a collection of executable units. Jobs can have one or more steps, each containing an executable unit. Job-Dependencies: Logical events or conditions that must be satisfied in order for jobs to be scheduled for execution. Predecessor and Successor.Job-Flow: A collection of two or more jobs related by job-dependencies.Schedule: The current active set of scheduled jobs and job-flows ready to be released for execution and monitored by the Master Scheduler.
ComponentsMaster Scheduler: Central software component that defines, modifies, stores job and job-flow definitions, dependencies, security information, events, and schedules. Provides automatic submission of executables, monitors status of all jobs and job-flows, provides alerts, controls starting, stopping, restarting of jobs and job-flows. Creates Audit trail of all events.Agent: A program that resides on each server that is controlled by the Master. The agent monitors jobs on its server, communicates with the Master about the status of jobs, and executes commands relayed by the Master.GUI Console: Console display for Administrators and Operators.Relational Database: The central repository used by the Master to store all job, job-flow, dependencies, security information, schedules and audit trails.
Job Scheduler Terminology and Components
Job Scheduler Terminology and Components (Cont.)
Agent
Master Scheduler
RDMS
Operator GUI console
Administrator
Agent
Agent
Logs
Administrator GUI console
Operator
Job-Flows
Job-Flows
Job-Flows
OPUS: Legacy Job Scheduler used for Satellite data processing
Developed by Space Telescope Science Institute (STScI ) in 1994 for HST Very fast and efficient pipeline processor. Powerful job-flow features and GUI consoleAdopted and customized for NOAA as a Job Scheduler on AIX p-series for migration from z/OS Mainframe (2005)OPUS requires replacement by a COTS Scheduler:
STScI no longer supports OPUSOPUS GUI written in Java has compatibility issues with latest version of Java
The Scheduler Study
Evaluated using two in-house Schedulers (NDE and STARS)Evaluated COTS Schedulers and vendors.
Scheduler Study Overview
In the beginning… there were 80 column Hollerith cards The term “batch” was derived from stacks of punched cards The first true Job Schedulers were people and card readers Therefore, originally Job Scheduling known as “batch processing” 1964 IBM OS/360 JCL presented first job dependency functions
Brief History of Job Scheduling
Early Mainframe era: Batch processing using card decks
Later Mainframe era: JCL on IBM Mainframes.
Jobs: Dependencies/Priorities/Queues/TOD-Calendar/Alerts/Reruns
Evolution of 3rd party vendor Mainframe Job Schedulers IBM Tivoli Workload Scheduler (TWS) CA 7 Mainframe Scheduler BMC Control-M ASG Zeke Scheduler
Unix systems: crons
Today: Job Scheduling has greatly matured, with a number of software vendors that have developed and marketed their Job Scheduler products for over 25 years
Brief History of Job Scheduling (Cont.)
Workload Automation Maturity Pyramid
Brief History of Job Scheduling (Cont.)
Gartner: The future automation model will be “IT Workload Automation Broker” (ITWAB)
Brief History of Job Scheduling (Cont.)
Scheduler Functions:
Create and edit jobs, job-flows, users, groups, schedules Calendar scheduling File-Watchers (Trigger file events) Job parameters within job-flows available Job Queues, Classes, Priority Virtual resources to control jobs and files High Availability / Failover capability Reliable and fast Scheduler database with open schema Scheduler commands to start, stop, suspend, resume, and restart jobs Alerts generated for job errors and sent to Operators (GUI console, email, cell) Reporting tools
GUI console: Windows, Web-based, Unix/Linux Flexible and configurable for Administrators and Operators, easy to use Graphical display of jobs and job-flows. Click and drag highly preferable Detailed display of job-flow and job status, dependencies, errors/alerts
Job Scheduler Functions
Architecture: Master-Agent vs. Peer-to-Peer
Job Scheduler Architectures
Architecture: Master – Agent
Job Scheduler Architectures (Cont.)
EMA Criteria and Methodology for Evaluating Job Schedulers
Research/Advisory Firms Job Scheduler Evaluations
EMA Radar Report for Job Schedulers 01/2010
Research/Advisory Firms Job Scheduler Evaluations (Cont.)
Forrester List of Evaluated Job Schedulers
November 2009 “Market Overview: Workload Automation, Q3 2009”Workload Automation Vendors
Research/Advisory Firms Job Scheduler Evaluations (Cont.)
Gartner Job Scheduler Magic Quadrant Report 04/2009
The Gartner Magic Quadrant is copyrighted (04/28/2009) by Gartner, Inc., and is reused with permission. The Magic Quadrant is a graphical representation of a marketplace at and for a specific time period. It depicts Gartner’s analysis of how certain vendors measure against criteria for that marketplace, as defined by Gartner. Gartner does not endorse any vendor, product or service depicted in the Magic Quadrant, and does not advise technology users to select only those vendors placed in the “Leaders” quadrant. The Magic Quadrant is intended solely as a research tool, and is not meant to be a specific guide to action. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
Source: Gartner, Inc.
Magic Quadrant for Job Scheduling
28 April 2009, Report Id Number: G00166953
Milind Govekar, Gartner Analyst
Research/Advisory Firms Job Scheduler Evaluations (Cont.)
Q/A
Additional Slides for reference…
18
Workload Automation Example
19
Future Growth of WLA and Job Scheduling
Forrester: Forecast for Growth of WLA and Job Scheduling
Evaluation of Job Schedulers and Vendors
Vendor Scheduler Product Name Open Source,
Freeware, COTS
ASCI ActiveBatch COTS
Absyss Visual TOM COTS
ActivMindz TaskForest Open Source
APM Software GmbH APX/PCC COTS
Apple XGRID COTS
Argent Job Scheduler COTS
ASG Zena for Distributed Systems COTS
ASG Zeke for z/OS Mainframes COTS
Automation Anywhere Automation Anywhere COTS
Ayehu eyeShare COTS
BMC Control-M COTS
Camellia Software Batch Job Server (BJS) COTS
CISCO Tidal Enterprise Scheduler (TES) COTS
Cluster Resources Maui Cluster Scheduler Open Source
Computer Associates AutoSys COTS
Evaluation of Job Schedulers and Vendors
Vendor Scheduler Product Name Open Source,
Freeware, COTS
Corporate Practical Solutions Bqplus COTS
Cronless.com cronless Open SourceDavid G. Walker php JobScheduler Free Ware
Exact JobBOSS COTSExspans Automan COTS
Flux Corp. Flux COTS
Grand Logic JobServer COTS
Help/Systems Robot/SCHEDULE Enterprise COTSHonico Batchman COTS
Hux Software HQ Scheduler COTS
IBMTivoli Workload Automation Suite
COTS
IndependIT BICsuite Scheduler Open Source
ISE EnterpriseSCHEDULE COTS
JOX IT JOX COTSLXI LXI Scheduler COTS
Evaluation of Job Schedulers and Vendors
Vendor Scheduler Product Name Open Source,
Freeware, COTS
MJTNet Macro Scheduler COTSMVP Systems JAMS COTSNeteject.com VisualCron COTS
Network Automation Automate7 COTSNguyen Anthuan SetCronJob Free Ware
OAR Team OAR Open SourceOpen Systems Mgmt Cosbatch COTS
OpsWise Automation Center COTSORSYP Dollar Universe COTS
OW2 Consortium ProActive Parallel Suite Open SourcePlatform Computing Load Sharing Facility (LSF) COTS
PTC Software PTC Scheduler COTSRedwood Software Cronacle COTS
ROC Software Maestro Scheduler COTSRuntime Design
AutomationNetwork Computer (NC) COTS
SMA OpCon/xps COTSSoftTree Tech 24x7 Scheduler COTS
Evaluation of Job Schedulers and Vendors
Vendor Scheduler Product Name Open Source,
Freeware, COTS
Sourceforge Jcrontab Open Source
Stonebranch Indesca COTS
Sun Microsystems N1 Grid Engine COTS
Sypsoft Job2Do COTS
Tenth Generation Job Scheduler for Windows COTS
Terracotta Quartz Enterprise Job Scheduler Open Source
The Coens Group Jobstream Open Source
UC4 Workload Automation Suite COTS
Univa UD Grid MP COTS
University of Warwick TITAN Grid Scheduler Open Source
University of Wisconsin Condor Open Source
Vinzant Global ECS (GECS) COTS
Xi Software GNUbatch Open Source
XuiS Software EnterpriseSchedule COTS