Technology Update - Dimensional Insight...Technology Update Mike Phillips, Technical Director Jamie Clark, Technical Director James Kirtley, Principal Software Engineer Dimensional

©2019 Dimensional Insight, Inc.

Technology Update

Mike Phillips, Technical Director

Jamie Clark, Technical Director

James Kirtley, Principal Software Engineer

Dimensional Insight User Group

2019


Diver Platform Architecture






Platform Map

1

2

4

Measure

Deploy

Prepare 3 Deliver


Part 1: Prepare

▪ Consistent approach

to the data lake

▪ Built-in

transformation

pipeline

▪ Intuitive analysis

tools for validation

1


2

Part 2: Measure

▪ Measure-centric

approach

▪ Designed for subject

matter expert

▪ Integrated

presentation &

development


3

Part 3: Deliver

▪ Designed for end user

▪ Practical self service

▪ Open architecture


4

Part 4: Deploy

Beginning to end


Part 1: Prepare


Part 1: Prepare

1


Diving In The Data Lake

▪ DiveLine server as Data Lake

– Extract data from many sources

– Store as Models, cBases, or text

▪ From there:

– Prepare for users

– Load into a Measure Factory


Preparation

▪ But, first:

– Clean up

– Combine

– Transform

▪ Integrator is still our most powerful ETL tool

▪ Spectre is catching up


Spectre – Build

▪ New build operation: Join

– Like Integrator’s join

– Left-outer (similar to lookup)

– Supports right-outer, inner, outer, for advanced join scenarios

▪ New build operation: Rotate

– Turns multiple columns into multiple rows

– Like Integrator’s rotate

▪ Improved memory usage

– In some cases


Spectre – Dive and cPlan

▪ New cPlan operation: Date rollup

– Create a new dimension by rolling up a date

– Like ProDiver’s date rollup feature

▪ New dive operation: Detail windows

– No summarization

– Like ProDiver’s detail dump feature

– Useful for back-end processing, to avoid merging records


Spectre – Calc updates

▪ on_previous_row() updated

– To enable detail-level cumulative calculations

▪ New calcs

– slope() and intercept()

– bit_get()

– format_datetime()

– fixed100()


Managing Complexity

▪ Powerful low level

tools

▪ Need a high level

tool to manage


Spectre Build UI


Spectre Build UI

▪ Same presentation style as Visual

Integrator/Builder.

▪ Helps you understand the whole

process


Table Editor

▪ Can now unlock

and modify

tables

▪ Includes:

– Sort

– Filter


▪ Raw Data Analysis

– Validation

– One-off queries

▪ ProDiver handles this

well

– But more on ProDiver

later

Raw Data Analysis with ProDiver


Part 1 Review: Prepare

▪ DiveLine as Data Lake

▪ ETL with Spectre

▪ New Spectre Features:

– Join, rotate, date rollup, detail windows, new calcs

▪ New Spectre Build UI

▪ New Table Editor Features

▪ Raw Data Analysis with ProDiver

▪ process


Coffee Break


Part 2: Measure


Platform Map

1

2

4

Measure

Deploy

Prepare 3 Deliver


2

Part 2: Measure

▪ Measure-centric

approach

▪ Designed for subject

matter expert

▪ Integrated

presentation &

development


Data Governance

▪ Data you can depend on

– Correct

– Rich

– Useful

▪ Requires data governance


Measure Factory - Governance

▪ Crucial to assessing

and improving

performance

▪ Ensuring relevant,

reliable and trusted

information

▪ “Single version of

the truth”

Accurate, consistent, and well understood measures


Measure Factory Overview

▪ High-level management of business logic

▪ Rules

– Record-level, no summarization

– Encode core business logic

▪ Measures

– User-visible, summarized

– Lots of metadata


Measure Factory Advantages

▪ Matrix Portlet

▪ Time series configured in portlets

▪ Measure Metadata

▪ Automatic analysis

▪ Assisted Analysis, Stamps, Self Service


Measure Factory UI

▪ Manage a complex measure factory

▪ Quickly find measures and rules

▪ Follow references

▪ Edit calculations and documentation


Measure Factory UI


Measure Factory UI


Measure Factory UI


Measure Factory UI

▪ Released in 7.1(1) with Rules and Measures pages

– Search, group, filter, sort, add, copy, delete…

– Links to referenced elements

▪ 7.1(4) added a Data Sets page

– Manage flag tables, lookups, plugins and links


Part 2 Review: Measure

▪ Data governance and automation, provided by the

Measure Factory

▪ Many “downstream” advantages

– Matrix Portlet, automatic analysis, Time Series, …

▪ New factory editor user interface


Part 3: Deliver


3

Part 3: Deliver

▪ Self Service

▪ Assisted Analysis

▪ Predictive Analytics

▪ Various Platforms

– Mobile

– DivePort Measures

and Analysis

– ProDiver


Delivery

▪ Diverse user community

▪ Requires flexible and

powerful delivery tools

▪ Reflected in DI’s mission


▪ Data Validation

▪ Raw Data Analysis

▪ Ad-Hoc Analysis of Managed/Governed Data

ProDiver


▪ Change Dimensions

in Multitabs

ProDiver – Dimension Switching


▪ New Toolbar design

▪ Better organization

▪ Larger, cleaner, easier to read

ProDiver - Toolbar


▪ Console Filtering

ProDiver – Console Filtering


▪ ProDiver Report Styles

ProDiver – Report Styles


Mobile Analytics with DiveTabAccess data any time, anywhere, on any device

▪ Multiple form-factors

– Smartphone

– Tablet

– Laptop

▪ Multiple platforms

– Android

– iOS

– Windows

▪ Online & offline access

▪ Dashboards, reports, maps, diving,

surveys, and more…


DiveTab DifferentiatorsDiving, volume, and airplane mode

Others DiveTab

Snorkel – shallowSnorkel – basically 2D, low volumeSnorkel – needs air, connected

Scuba – diving deepScuba – 3D, high volumeScuba – has a tank, not connected


DiveTab Changes

▪ Offline Exploration

▪ Logging User Activity

▪ Surveys

– Location-aware questions

– Questions filtered by other questions

▪ Improved Excel Export


DivePort

▪ Dashboards

▪ Scorecards

▪ Governed Analysis

▪ 7.1 Changes

– Portlet Gallery

– Skin updates

– Indicator style

– New QuickViews

– New Charts


DivePort – Portlet Gallery

▪ New way to choose a portlet to add

▪ Quick Search feature

▪ Categories

▪ Sort option


DivePort – Skin Changes

▪ Improvements to the default skin

– New chart color palette

– New colors in dialogs and navigation elements

▪ Previous default is still available as “7.0 Default”


DivePort – Indicator Changes

▪ Cleaner visual style

▪ New Indicators:

– Fill Gauge

– Circular Gauge


DivePort – QuickView Changes

▪ New “Flex” style, with lots of options

– Can be collapsible like a pulldown

– Can be searchable like a combobox

– Can select multiple items like a picklist


DivePort – Chart Changes

▪ New chart types

– Stringline

– Calendar Heatmatrix

– Trellis

– Mosaic

▪ Custom hover


DivePort – Measures/Diver Portlet Changes

▪ The Measures Portlet can now dive

– Separate Diver Portlet no longer needed

– New dialog style

– Also, mouse wheel support


Self Service

▪ Empower and encourage the end user

– It has to be easy

– A data lake is not easy

▪ Measure Factory

– Automation

– Metadata

– All data in one place

– Trusted


▪ Preconfigured self-service visualizations adapt to a specified

measure

▪ Integrated analysis built into measures – automatically

Self-Service - Stamps


Self Service Demo


▪ What is a stamp?

– A collection of portlets, pre-

configured and arranged for display

– Reusable with different measures

▪ Where do I get them?

– Default stamps provided by DI

– DI Applications provide additional

application-specific stamps

– Site admins can add their own

custom stamps

Self-Service – Stamp Configuration


▪ The Stamp Portlet renders a stamp on a page

– Simple portlet configuration

– Edit the portlet to change the measure

▪ For self-service, each user has their own page

– New “Stamp Grid” layout

– Drag and drop stamps in your own playground

Self-Service – Stamp Usage


Agile Exploration

Unlimited diving without pre-configured dive paths or hierarchies

Measures Dimensions Summaries

Dimension ValuesSuppliers

▪ Dashboard Analysis

Example:

– 7 measures

– 69 dimensions

– 51 different suppliers

– Assuming you dive only

3 deep, the total number

of permutations is

114,428,496

▪ Completely automated


▪ Too many possibilities

– Let the software help you find interesting

data

▪ With Assisted Analysis:

– Software looks ahead

– Gives you suggestions

– Rates them on a scale of 1-9

– You take it from there

Assisted Analysis


▪ New Feature in Analysis Portlet

▪ Three modes– Assisted Analysis

• Lists dimensions by importance

• Lists values in those dimensions by importance

– Standard dive table

• For manual control

– Measure history chart

Assisted Analysis – Usage


Assisted Analysis Demo


▪ What makes a value “interesting”

– Must be something a computer can detect

– Doesn’t have to be perfect

▪ Default algorithm

– “3-sigma” standard deviation analysis

Assisted Analysis – Algorithms


▪ This is just the beginning

▪ Future ideas

– Look for control-chart-style rule violations

– Predictive analytics

Assisted Analysis – Algorithms


Predictive Analytics and Data Science

▪ Open Architecture allows use of advanced third-party tools


▪ Use external tools for statistical analysis and machine

learning

▪ Visualize within dashboards

– Regressions

– Clustering

– Outlier

detection

– Time series

forecasting

Predictive Analytics


Part 3 Review: Deliver

▪ ProDiver– Change dimension, console filters, updated UI elements

▪ DiveTab

▪ DivePort– Portlet gallery, new QuickViews, new charts, indicator/measures

changes

▪ Self Service– Stamps, stamp grid layout, user pages

▪ Assisted Analysis

▪ Predictive Analytics


Part 4: Deploy


4

Part 4: Deploy

Beginning to end


▪ DI’s Client/Server Distributed IDE

▪ One-Stop Shopping for all development needs

▪ Server Configuration (users, groups, access control)

▪ Server Maintenance (active users, processes)

▪ Allows multiple simultaneous users

▪ Version Control/File History

Workbench Review


New In Workbench 7.1

▪ View Switcher

▪ Process Manager

▪ Spectre Build UI

▪ Expanded Table Editor

▪ Connection Manager/Quick Launch dialog

▪ Measure Factory UI


View Switcher

▪ Simplifies multiple different views of the same file

▪ Easily switch between text and GUI views

▪ New “Run History” control for Process Manager


Process Manager

▪ Handles Integrator, Builder,

Spectre Processes

▪ Keeps track of all processes

running on the server

– Who launched them?

– How much

memory/processor time are

they using?

▪ View Logs -

Success/Failure, Elapsed

Time, output, etc.


Connection Manager/Quick Launch

▪ Consistent Connection

Manager UI

▪ New “Quick Launch” dialog

makes connecting to select

servers quicker.


Open Architecture

Analysts can use their tool of choice


ODBC

▪ New DI-ODBC Driver– Connect third-party applications to DiveLine

– Use SQL to query a Measure Factory, cBase, cPlan, or dive script

▪ Examples– Synchronize data into an Excel

sheet

– Use the Spectre engine to feed

an alternative graphing tool

– Dump cBase data to import

into another application


Part 4 Review: Deploy

▪ Workbench

– View switcher

– Process Manager

– Quick Launch

– Connection Manager

▪ DI-ODBC

– Connect to third-party tools


Future Plans


Platform Map

1

2

4

Measure

Deploy

Prepare 3 Deliver


Prepare: the Future

1 Prepare


Future: Spectre

▪ New system for analyzing the

whole build/dive process

– Reduce peak memory usage

– Share results of some operations

– Cache intermediate results

1


Future: Upload Portlet

▪ Drag & Drop file upload

▪ Text file, Excel sheets, images, docs…

▪ “Required columns” feature

▪ Custom validation and processing

1


Measure: the Future

2 Measure


Future: Measure Factory

▪ Track Data from Data Sources to Factory

– Measure Factory can track a measure back to the factory

inputs

– Extend metadata farther back

– To original data sources

▪ Multi-application support

– Techniques exist to combine applications in a factory

– Improve support for this

– Integrate it into the graphical interface



▪ Easier transition to factory

– Tomorrow we’ll discuss migrating to a factory

– Make this even easier

▪ Factory Editor UI Updates

– Analyses

– Factory settings


Deliver: the Future

3 Deliver


Future: DivePort

▪ Chart Animations

– Animations for (many) charts:

• “load” animation

• “change” animation

– Almost ready

▪ Map Filter Feature

– Filter portlets CrossLink-style by zooming into a map

– Almost ready


Future: DivePort

▪ More Charts

– Control Charts

▪ Slanted tick labels

– In testing


Future: DiveTab

▪ Self Service in DiveTab

– Add stamps to DiveTab

– Let users create their own pages

– Sync them across devices


Future: Assisted Analysis

▪ Add more algorithms to Assisted Analysis

– Statistical

– Predictive

– Control-chart

– And more


Future: ProDiver

▪ More interface changes

– Improve usability

▪ Measure Factory Integration

– Measure categories

– Show metadata


Deploy: the Future

4 Deploy


Future: User Management

▪ Improved Active Directory integration

– Connect to AD Roles

▪ Authentication standards

– Multi-Factor Authentication (MFA)

– OATH, FIDO2 / WebAuthn



▪ Limiting Measures selection

– Simplify for specialized users

▪ Speed up Factory builds



Documents

Technology Update - Dimensional Insight...Technology Update Mike Phillips, Technical Director Jamie Clark, Technical Director James Kirtley, Principal Software Engineer Dimensional