Embed Size (px)
Citation preview
The Metadata Tool Making ODS Metadata More Useful to End Users
Meredith Pritchard
BI Developer/Analyst, Denison University
About Denison
Founded in 1831 Formerly known as Granville College (until 1853) Liberal Arts college (38% acceptance rate) 2,250 students (2014-15 - 43% male, 57%
female, 36% multicultural, 20% First Generation) 226 Full Time Faculty Fall and Spring semesters Residential campus Located in Granville, OH – just 25 miles east of
Columbus Source: Wikipedia
ERP and Reporting Environment
Banner ◦ Live in 1997 ◦ Version: 8 ◦ Currently implementing XE in Test Environment
ODS ◦ 8.4.1 ◦ Materialized Views
Cognos ◦ 10.2.2 ◦ Report Studio = 10 IT ◦ Query Studio and CWA = 50 End Users ◦ Consumers = 200+ Faculty and Staff
Oracle 11.2.0.4 ◦ Linux (virtualized)
Enterprise Apps Team
1 Director
1 DBA
4 Programmer Analysts
2 BI Analysts
1 Portal Admin / Web Developer
1 Integration Specialist / Web Developer
Most Popular Scenarios for utilizing Delivered Metadata
End Users ◦ I want to create a Query in Cognos which includes these Banner fields. What package do I use?
Report Writers ◦ Translating a business spec into a report spec. Does this Cognos Data Item map to the Banner field that the customer needs?
Example Delivered Metadata
Delivered Metadata Downfalls & Project Goal
Delivered Metadata Downfalls ◦ Difficult for End Users to navigate
◦ Like drinking from a fire hose
◦ Incomplete - does not map to Cognos Packages
Goal ◦ Create useable metadata for End Users
Solution Overview
1. Scrape the Cognos FWM model xml (sghe_ods_bv_model.xml) for the lineage
2. Join scraped model data with WMT table data from IA_ADMIN schema
3. Create a table (DU_Metadata) with complete lineage data
4. Model the DU_Metadata table in FWM 5. Create a report that uses DU_Metadata table to
show lineage – mapping a Banner field to all possible Cognos Packages or vice versa
6. Publish Report to all Folders & Create Job Aid
Solution Detail – Step 1: Scrape the Cognos Model
This process uses the delivered FWM model (a large, ugly xml file - sghe_ods_bv_model.xml) and the WMT Tables from the IA_ADMIN schema.
Credit for this process goes to our DBA Ninja, Doug Davenport.
Solution Detail – Step 1: Scrape the Cognos Model
1. Used tool (xmllint --format) to reformat the xml file so he could inspect the model to determine patterns within model
2. Create an xslt query (query language for xml) to document the various patterns (heirarchies) of the xml data. Example of 1 of the patterns:
<xsl:text>"</xsl:text> <xsl:value-of select="../../../../cognos:name"/> <xsl:text>","</xsl:text> <!-- Package Name --> <xsl:value-of select="../../../cognos:name"/> <xsl:text>","</xsl:text> <!-- Query Subject --> <xsl:value-of select="../cognos:name"/> <xsl:text>","</xsl:text> <!-- Query Item --> <xsl:value-of select="cognos:name"/> <xsl:text>","</xsl:text> <!-- Expression --> <xsl:value-of select="normalize-space(cognos:expression)"/> <xsl:text>"</xsl:text> <!-- Output a new line --> <xsl:text> </xsl:text> </xsl:for-each>
Solution Detail – Step 1: Scrape the Cognos Model
3. Create 1st shell script that:
Checks out model xml from git repository
Runs the xslt query (from step 2) against the xml model using xsltproc (command line utility)
Creates du_query_item.csv which becomes the basis of an Oracle External table
Solution Detail – Step 1: Scrape the Cognos Model
DU_QUERY_ITEM
Solution Detail – Step 2: Join metadata to model data
3. Create 2nd shell script (du_metadata.shl) that:
Uses PL/SQL to loop through the du_query_item external table and supplement each row with data from the WMT tables from IA_ADMIN, then writes a row to an oracle table (du_metadata).
Solution Detail – Step 2: Join metadata to model data
Snippet of DU_METADATA.SHL: DECLARE l_rec1 du_ods.du_query_item%ROWTYPE; l_rec2 ia_admin.wmt_source_to_target_map%ROWTYPE; l_rec3 ia_admin.wmt_target_column%ROWTYPE; l_rec4 ia_admin.wmt_target%ROWTYPE; l_rec5 ia_admin.wmt_source_column%ROWTYPE; l_rec6 ia_admin.wmt_source%ROWTYPE; l_rec7 ia_admin.wmt_subject_area%ROWTYPE; l_tbl varchar(32); BEGIN FOR l_rec1 IN (SELECT * FROM du_ods.du_query_item) LOOP BEGIN l_tbl := 'wmt_source_to_target_map'; SELECT * INTO l_rec2 FROM ia_admin.wmt_source_to_target_map WHERE TARGET_TYPE = 'REPORTING VIEW' AND TARGET_NAME = UPPER(REPLACE(l_rec1.query_view, ' ', '_')) AND TARGET_COLUMN_NAME = UPPER(REPLACE(l_rec1.query_column, ' ', '_')) AND ROWNUM = 1; -- l_tbl := 'wmt_target_column'; SELECT * INTO l_rec3 FROM ia_admin.wmt_target_column WHERE TARGET_TYPE = 'REPORTING VIEW' AND TARGET_NAME = UPPER(REPLACE(l_rec1.query_view, ' ', '_')) AND TARGET_COLUMN_NAME = UPPER(REPLACE(l_rec1.query_column, ' ', '_')) AND ROWNUM = 1; …
Solution Detail – Step 3 : Create Table
DU_Metadata
Solution Detail – Step 3 : Create Table
DU_Metadata
Solution Detail – Step 3 : Create Table
DU_Metadata
Solution Detail – Step 3 : Create Table
DU_Metadata
Solution Detail – Step 3 : Create Table
DU_Metadata
Solution Detail – Step 3 : Create Table
DU_Metadata
Solution Detail – Step 3 : Create Table
DU_Metadata
Solution Detail – Step 3 : Create Table
DU_Metadata
Solution Detail – Step 4 : Model DU_Metadata in FWM
Solution Detail – Step 4 : Model DU_Metadata in FWM
Solution Detail – Step 5 : Create Report
Prompt Page
Solution Detail – Step 5 : Create Report
Results and Re-Prompt Page
Solution Detail – Step 5 : Create Report
Demo
Solution Detail – Step 6 : Publish Report & Create Job Aid
1. Report is stored in a limited access folder
2. Report views are published to all Functional Area Professional Reports folders
3. Promote this report to top of all Professional Reports folders
4. Create job aid for users on how to do a lookup on a Banner form to identify field name. Then how to use the Metadata Tool (report) to find location of that field in ODS
Solution Detail – Step 6 : Publish Report & Create Job Aid
Solution Detail – Step 6 : Publish Report & Create Job Aid
Solution Detail – Step 6 : Publish Report & Create Job Aid
Solution Detail – Step 6 : Publish Report & Create Job Aid
Solution Detail – Step 6 : Publish Report & Create Job Aid
Solution Detail – Step 6 : Publish Report & Create Job Aid
Caveats & Disclaimers
The join between the Cognos metadata and the ODS metadata is an inner join. Of the approximate 60,000 items in the model, roughly 4000 are being excluded and written to the exception logs
A sizeable portion are custom query subjects/reporting views (Examples: Constituent Spouse_DU, Advancement Custom Calcs_DU).
Another sizeable chunk are LOV and Presentation view (level of model) query subjects.
Some are delivered views that are created within the model (not in ODS) and therefore no metadata exists. (Example: Telephone Current – in ODS 8.3)
Lastly, there is one data item (ID) from Person_Address that is on the log that is referred to as 1 thing in the database, but another in the model (ID1). I attempted to resolve by adding an ID1 to the metadata, but then it threw an error on the Health Check saying there was an item in the metadata but missing from the view. So the solution would be to rename the field in the model, but at the risk of breaking reports that may be using that field. At this time, we choose not to address this.
Caveats & Disclaimers
Therefore, we expect and accept that there may be the occasion when this tool fails to provide the data requested. We believe it to be 90-something-ish% effective. However, we have not yet faced the scenario when an End User reported that they were unable to determine the lineage of an item using this tool.
The scraping process is somewhat fragile and not guaranteed to be 100% effective.
We have only used this with ODS versions 8.3 and 8.4.1. This is a run-on-demand job. Typically we run after an upgrade. We are still left with the question of how to incorporate our
custom view metadata into this tool. We haven’t yet spent the time/resources to determine how we can stuff the metadata from those custom views into the model so that we can scrape it along with the delivered metadata. Or if that is even the right approach.
Possible Enhancements
Access Banner Table and column descriptions (They are not currently coming through in the metadata, but could be accessed directly from Banner.)
Add sort order from Query Subject in Query Studio.
Stuff lineage back into the model under ‘Description’ in the properties – use 2nd language?
Add to daily refresh
The End
Questions?
