iWay Data Profiler

Anand Maheshwari

Product Highlights

5.0 GA

iWay Data Profiler

1

What is iWay Data Profiler

2

iWay Data Profiler provides the ability to look at data quality from a business user perspective with dashboard templates to view, monitor, compare, and report by leveraging the turnkey business intelligence technology through semantic analysis, complex business rules and data drill-down.

Features

1. Ability to profile data against a set of standard (pre-defined) data quality rules and metrics

2. Ability to create custom organization specific data quality indicators3. Advanced comparison of data profiling and quality indicators4. Ability to create and execute data quality projects against varying payloads5. Ability to rollup data with drill through6. Advanced administration capability7. Advanced auditing capability8. Easier migration from earlier versions9. Ability to track background processes for faster troubleshooting and status10. Use of current standards like JQuery for enhanced user experience11. Support for all leading database platforms12. Support for wide variety of OS platforms13. Support for German, French, Spanish and English locales14. Increased performance through several tuning efforts

3

4

Features 4.1 5.0

Create profile with drill-through

Export profile report to: AHTML and PDF format AHTML

PDF

XLS ×

Email profile using SSL

Create profile report for: Data Analyses

Business Rules

Primary Key

Foreign Key

Summary

Hierarchical folders ×

Uploading profile in background ×

View process result ×

Rollup profile with drill-through ×

Profile file size 100MB 300MB

1. Ability to profile data against a set of standard (pre-defined) data quality rules and metrics - Comparison

1. Ability to profile data against a set of standard (pre-defined) data quality rules and metrics – New Features

5

• Uploading profile in background

• View result to display uploading progress

• Export profile report in XLS format

6

2. Ability to create custom organization specific data quality indicators – Comparison

Features 4.1 5.0

Create and view DQIs report

Export PDF report

Folder Hierarchical division ×

Comparison Across Expression ×

Visualization for success and failure counts ×

More user interaction using AHTML Format ×

2. Ability to create custom organization specific data quality indicators – New Features

7

Easy to understand the data quality indicators through multiple reports

Flexibility to compare the expressions View Result to display progress for the background processes


8

Easy to understand the data quality indicators through multiple reports


9

Flexibility to compare the expressions


10

View Result to display uploading progress

11

Features 4.1 5.0

Quick compare of profiles/DQIs

Create Dataset & Template

Graphical representation of profile/DQI compare

Default Graph Line 3D Bar

Displaying table report with visualization ×

Hierarchical folders division ×

3. Advanced comparison of data profiling and quality indicators - Comparison

3. Advanced comparison of data profiling and quality indicators – New Features

12

Additional table report and chart type

• Display of both success and failure count and percentages

• Informative table report display with charts

4. Advanced comparison of data profiling and quality indicators – New Features

13

New design of data displayed in dataset table, for Profile & DQI

Modified default report as 3D Bar

4. Ability to create and execute data quality projects against varying payloads – New Features

14

• Data Quality Projects for Profiles/DQIs enables running a DQC plan for generating a Data Profile/DQI and loading into database repository• Create data quality project• Manage data quality project

• Reusability of plan - The same plan can be executed against different inputs• View Result for different level of progress


15

Create project – option to upload Profiling plans and DQI plans

Plan can be with drill-through and also with Rollups


16

Manage project


17

Reusability of plan : The same plan can be executed against different inputs


18

Note:

Plan is supported with TXT and CSV Input, Other Input in DQC like XML, Database is not supported in the current version

Output can only be profile steps (or DQI steps depending on the type of DQ project)

The following input settings of the input file can be modified without DQC GUI:• Number Of Lines In Header • Field Separator • Line Separator • String Qualifier • Encoding


19

View Result for different level of progress

5. Ability to rollup data with drill through – New Features

20

Roll up is a part of iWay Data Profiler that enables a user to profile view a specific subset of a data.

Rollups for: Data Analysis Primary Key Business Rules.

5. Ability to rollup data with drill through – New Features

21

Enabled drill-through for the subset of data

6. Advanced administration capability – Comparison

22

Features 4.1 5.0

Create Roles, Groups and Users

LDAP User Authentication and LDAP Role Authorization

Option to create, delete and manage folders ×

Group and Role details can be viewed while creating a user ×

Groups have access to one or more Virtual Folders ×

Base Role : Admin, Power, Technical and Analytical with different privileges

×

iSM Tracking privilege ×

DQ Project privilege ×

Rollup privilege ×

6. Advanced administration capability – New Features

23

• Hierarchical folder structure to manage profiles, DQIs, datasets, templates Grouping related things together based on the user choice Sharing and easy to compare Created a group called “Admin” and linked all folders to it Folder structure for a user

o Private o My Groupo Customo Public

• Created default base roles: admin, power, technical and analytical with varying authorizations


24

Manage Folders


25

Manage Roles


26

Manage Groups


27

Manage Users

7. Advanced auditing capability – New Features

28

Audit (Tracking user name, IP address, login time and logout time) with different databases

• Enhanced user security

• Filter audit information

• Track log in and log out time in iDP with IP Address

• Track invalid login & iDP server manager login


29

Audit information for a user


30

Filter audit information

8. Easier migration from earlier versions – New Features

31

IncludesUpgrade script is for migrating from the older version to newer version

4.1 5.0Available for:

MySql MSSQL Oracle

There is no loss of data exists in database “idp_data”

Does not includeSynonym creationDrill-through adapter creation in WebFOCUS

9. Ability to track background processes for faster troubleshooting and status – New Feature

32

Interface for the user to monitor the back-end process Multilingual support for view process result – Available for Profile, DQI, DQ

Project and Custom Reports

iWay Data Profiler : Architecture

33

iWay Data Profiler - Component Architecture

34

Benefits

Web enabled thereby single deployment and access anywhere Aids in the identification of data quality issues Expertise and support of a mature business intelligence product –

WebFOCUS Data quality comparison over time Documents and audits quality-control processes for review Streamlines offline maintenance activities to ensure optimum performance Share the reports using a secured connection Mobile support

35

Anand Maheshwari

Thank You !

5.0 GA

iWay Data Profiler

36

Appendix – Additional Features

37

Multilingual support

38

iWay Data Profiler supports multi-languages English German Spanish French

Note: ISM Tracking Edit page language will change corresponding to the language choose in ISM configured not based on the IDP chosen language.

Performance tuning

39

JAVA code has been restructured for better performance

• Complete recoding using XPATH and transaction based multiple coding being undertaken as processing of complicated Rollups found to crash the JVM

• The maximum size of the Profile XML file, which can be uploaded into DP, has been increased from 1MB to 300 MB

Revamped iWay Data Profiler with JQuery

40

JQuery Functionalities• Enhance look and feel• Styling• Cross browser compatibility• Support all mobile devices• Grouping related things together• Sharing• Easy to compare• Fast loading• Alerts and alignment • Expand All/ Collapse All functionality• More/ Less functionality• Pagination• Homepage carousel – to show case different graphs of iDP• Landing page accordion layout – to display different iDP modules

Modified both Windows and Linux installer

41

Updated both Windows and Linux Installer scripts for DQ Project to get the Drill-through DB credential from the installer

Installer is available in English, German, French, Spanish Same Linux build is supported for AIX platform

Backward Compatibility with iWay Data Quality Center

42

Modified iDP for Profiling DQI

Support across versions DQC 7.0.1 & higher Input data Reports