View
41
Download
0
Category
Preview:
Citation preview
1© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Extending the Admin UI: Proactively Preventing Poor BPRA Load/Refresh Performance via Custom Utilities
Eric Boyce - DBABryan Mack – BI Development Team Lead
April 14, 2015Session 11951
2© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Session Rules of Etiquette
• Please silence your cell phone/pager/purse dog
• If you must leave the session early, please do so as discreetly as possible
• Please avoid side conversation during the session
• Please hold questions for the end of the presentation
Thank you for your cooperation!
3© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Session rules of etiquette
Bryan L. Mackbryan.mack@cccs.edu
Higher-Ed Employment History
• Colorado School of Mines, Asst. Director of Advancement Services 2002-2012
• Colorado Community College System, Team Lead/Data Warehouse Developer, 2012-2015
4© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Session rules of etiquette
Eric Boyceeric.boyce@cccs.edu
Higher-Ed Employment History
• University of Northern Colorado, BPRA Database Administrator 2009-2012
• Colorado Community College System, BI Sr. Oracle Database Administrator, 2012-2015
5© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Purpose
• To create an understanding of how simple it can be to add desired functionality to the BPRA Administration Interface. Further, we’ll step through two scenarios whereby the Colorado Community College System benefited from augmenting the BPRA Administration Interface’s functionality.
6© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
1 Problem I – Inconsistent ODS Refresh Times
2 Potential Fixes – Baselines, Profiles, Statistics
3 Implementing a Solution – Stats Gathering
4 Problem II – BPRA Batch Window Utilization
5 Implementing a Solution – Chain Builder
Agenda
7© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Problem I – Inconsistent ODS Refresh Times
8© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Inconsistent ODS Refresh Times
• Greeted with this in the Morning:
9© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Why the Poor Performance
Root Cause: Stale Statistics
● Over 4,300 daily batch jobs in CCCS’s Banner environment (AppWorx)● Due to process ordering and contingencies batch execution times are dynamic● Changing Maintenance Window for statistical collection ineffective
10© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Bad Stats Example
NUM_ROWS – Updated by stats collection
11© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Potential Fixes – Baselines, Profiles, Statistics
12© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Baselines, Profiles, Statistics
● There is nothing wrong with using baselines, profiles, or dynamic sampling to achieve performance improvements
● We wanted a tool that was simple, yet effective, in improving/maintaining ODS performance● We needed something that could handle Ellucian upgrades without
adversely affecting performancei.e. Not reliant upon sql_id values
● We know the problem is poor statistics. No need to put cart in front of the horse by creating baselines or profiles
13© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
When Should Statistics Be Collected?
According to Oracle
Source: http://docs.oracle.com/cd/E25054_01/server.1111/e16638/stats.htm#i41884
14© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Requirements
● Collect statistics on change tables, composite tables, and indexes just prior to the execution of a mapping
● Must be simple interface to understand and utilize… otherwise nobody will use it
● Must have ability to ascertain mappings’ objects dependencies
● Should have the ability to handle various DBMS_STATS parameters
15© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Implementing a Solution – Stats Gathering
16© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Identification
Identify exact locations where mappings are executed:
17© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Digging Deeper
● Matching String Literals from Control Report to MGKMAP Body
18© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Verification
Query against all_source
19© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Before we go further…
Ellucian’s stance on the modification of baseline code:
-Any customizations to baseline code are NOT supported-
20© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Customizing Baseline Code
IMPORTANT: Assure that the baseline scripts are saved off and available. If an error is encountered you can always revert to the functional baseline version.
21© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Testing New MGKMAP
● It worked! We now know where the stats collection should take place● We also have the mapping object name
22© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Creating a Package to Gather Statistics
PACKAGE: IA_ADMIN.CCCS_GATHER_ODS_STATS
PROCEDURES: p_gather_mapping_stats (mapping_name, estimate %) p_gather_composite_tab_stats(mapping_name, estimate %) p_gather_srce_and_change_stats(composite_view, estimate %) FUNCTIONS: f_get_stats_check (mapping_name)
23© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Creating a Package to Gather Statistics
p_gather_mapping_stats 1ST – Gather Composite Table Stats
- Determines Composite Table using DBA_DEPENDENCIES
24© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Creating a Package to Gather Statistics
p_gather_mapping_stats 2nd – Evaluate Mapping Type
- If it is a DELETE mapping, gather statistics on change table- Determines Change Table using DBA_DEPENDENCIES
25© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Creating a Package to Gather Statistics
p_gather_mapping_stats 3rd – If mapping is not a DELETE, then it must be an UPDATE or LOAD
mapping. - Determines Composite View and dependent tables
26© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Creating a Package to Gather Statistics
p_gather_mapping_stats 3rd (cont.)– If composite view found, call p_gather_srce_and_change_stats - Loops through source tables and collects stats
27© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Creating a Package to Gather Statistics
f_get_stats_check● Called from MGKMAP prior to any collection of stats
● Function checks to see if statistics should be collected for a particular mapping
● Allows for control as to which mappings collect statistics
● Uses a custom MTVPARM record
28© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Remember when…
● We want to collect stats on this mapping prior to execution to prevent this from happening in the future
29© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Create MTVPARM Parameter
● Beneficial to store the parameter values in MTVPARM as they’re easily accessible via the Admin UI
30© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Once again…
● Once again, we have bad statistics.
31© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Control Report – With Stats Collection
32© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Control Report – No Stats Collection
● If there is no MTVPARM record for a mapping, the f_get_stats_check will evaluate to FALSE and statistical collection is simply ignored.
33© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Control Report – Stats Collection on UPDATE
34© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Problem II - BPRA Batch Window Utilization
35© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Batch Window
● CCCS’s BPRA environment has a batch window from 00:00 – 07:30.
● ODS refreshes are executed concurrently as REFRESH_ALL takes too long.
● Implementing EDW and SRP caused overruns(52 Hours initially, cut down to 3 hours after in-house tuning)
● Varied ODS refresh times made it difficult to predict when to schedule the EDW Operational Refresh Job
36© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Need for Contingency
● SRP reliant upon ODS Student, ODS Finaid, and ODS General refresh completion
● Wanted a way to maximize our batch window by not allowing time between jobs.
● Also wanted the ability to run concurrent jobs
37© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Oracle Chains
“A chain is a named series of tasks that are linked together for a combined objective. Chains are the means by which you can implement dependency based scheduling, in which jobs are started depending on the outcomes of one or more previous jobs.”
Source: http://docs.oracle.com/cd/B28359_01/server.111/b28310/scheduse009.htm#ADMIN10021
38© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Oracle Chains
Chains can be thought of as a series of steps with defined rules as to when a step should be performed. A step is associated with a task or action.
Step Action When to start-------------------------------------------------------------------------------------------------------
Step 1 - Choose Bread TRUE (Immediately)Step 2 - Add Condiments
Step 1 CompletedStep 3 - Add Meat/Veggies Step 2 CompletedStep 4 - Assemble
Step 3 Completed Step 5 - Eat Sandwich Step 4 Completed
Step 6 - Go to Sleep (End) Step 5 Completed
39© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
In our Case
Step Action - Program When to start - Rule-------------------------------------------------------------------------------------------------------
Step 1 - Refresh Student TRUE (Immediately)
Step 2 - Refresh FinaidTRUE (Immediately)
Step 3 - Refresh General TRUE (Immediately)
Step 4 - Refresh Custom MViews Step 1 CompletedStep 5 - Refresh EDW Operational Steps 1, 2, 3,
4 Succeeded
40© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Defining Actions via Programs
● Oracle Scheduler Program Objects define an action for a Chain step● Can be used to abstract a PL/SQL Block, Stored Procedure, or an
executable
41© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Building Programs
● So couldn’t we just put these Refreshes into Programs using PL/SQL blocks?
DECLARE JOB_NUM NUMBER;
BEGIN SELECT sys.jobseq.nextval INTO JOB_NUM from dual; mgkmap.P_RunETLMapSlots(9,JOB_NUM,'REFRESH_FINAID',NULL, '');
END;
42© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Creation of a Program
● Could be done in Enterprise Manager but can be tedious when doing a large quantity.
43© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Implementing a Solution – Chain Builder
•
44© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Sample Admin UI Job Script
● Ellucian delivers a sample Admin UI job script in the ODS installation code tree:
ods\odsdevl\ods\ia_admin\dbscripts\utility_scripts\sample_job.sql
● Serves as a good reference for the necessary pieces when creating a custom job.
45© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
INSTALLED PROCESS Parameter
● Creation of an INSTALLED PROCESS Parameter creates a link in the Admin UI’s ‘Schedule a Process’ Menu
46© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
JOB Parameter
● Creation of a JOB Parameter links the Installed Process to a Packaged Procedure
47© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Basic Packaged Procedure Header
● Main procedure of sample job noted earlier
● Main procedure of CCCS_CHAIN_ADMIN
48© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Questions…Questions…Questions…
● Before we go any further we have to ask ourselves:
1. How many steps should the chain builder handle?2. How many rules will we need?3. How are the rules created?4. How can we monitor progress as chain is executed?5. How do we avoid having objects of the same name?
49© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
First Question
1. How many steps should the chain builder handle?A: CCCS chose to start with 7. This was the fewest number of
steps we would need to create a chain that could handle our EDW/SRP requirements.
Note: It was decided that although CCCS would start with 7 steps, the utility needed to be flexible enough to easily handle more steps in the future.
50© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Second Question
2. How many rules will we need? A: We need the number of steps decided upon +1
- This is because there is a final rule that states when all steps are finished, the chain is has completed–
- Additionally, the final rule need not be a parameter for a user to control as this could cause the chain to stall.
51© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Creating Parameters
52© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Rules and Steps
7 Rules and 7 Steps
53© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Additional Parameter
● We also wanted to allow the user to specify their own Chain Name
54© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
So How’s it Look
55© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Parameter Handling
● We use tables to store the values we’d like to display for parameter values. The ‘SELECT’ parameter type simply displays the results of a query.
56© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Parameter Handling (cont.)
Example.
57© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Translation Table
● We use the ia_admin.cccs_chain_prog_xwalk table to map Display names to actual object names. Further, the SEQUENCE_NUMBER
gives us a means of easily controlling, which processes are displayed in Admin UI.
58© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Rules Table
Rules possibilities are also stored in a table.
59© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Admin UI Submission
60© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
How Parameters are Passed
● The job generated contains all of the parameters inputted, in a pipe delimited format.
● Knowing all the parameters, we can now build a chain
61© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Parsing Parameters
● All parameters come into the parms input parameter
● Use mgkutil.F_GetParameterValue to determine what the inputted values are
62© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Building Chain
● After validation that the job has basic parameters in place, we verify that no other chains of the same name exist. If they do, they are removed and an entry noting this is made in the control report. Following this step, an empty chain is created
63© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Building Chain Steps
● Once the chain is built, we add the steps for it● Display names for programs are parsed to their actual object names● Using a counter variable we loop through all the programs creating chain steps along the way.
64© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Building Chain Rules
● Chain Rules are built in a similar manner as the Chain Steps with a loop and a counter variable.● There is one final rule that is built. It is the summation of all steps and informs Oracle that the chain is complete.
65© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Creating Scheduler Job
● We’ve now created a chain, added steps to it, and added rules to it.
● The only thing left to do now is to submit it.
66© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Checking Mechanism
● As we wanted to know the status of the scheduler job, we created a self checking mechanism● Every five minutes the chain run is evaluated and it’s status is reported in the control report
67© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Final Control Report
68© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Adding More Steps in the Future
● To add additional steps, all one needs to do is create additional PROGRAM and RULE parameters following the naming convention laid forth● Additionally, rules must be added to cccs_rule_possibilities or
whatever table is chosen to store the RULE literals
69© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Summary
With an understanding of how the Admin UI works, a developer can easily create utilities/functionality to augment what is already a great product delivered by Ellucian
70© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Questions & Answers
71© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Thank you!
Eric Boyce - DBABryan Mack – BI Development Team Lead
Please complete the online session evaluation form.Session 11951
Recommended