2018-09-25 AI in Spotfire X Inside and Out MOC Final1 · AI in Spotfire X: Inside and Out This document (including, without limitation, any product roadmap or statement of direction

Dan RopeAndrew BerridgeGaia PaoliniMichael O’Connell

September, 2018

AI in Spotfire X: Inside and Out

This document (including, without limitation, any product roadmap or statement of direction data) illustrates the planned testing, release and availability dates for TIBCO products and services. This document is provided for informational purposes only and its contents are subject to change without notice. TIBCO makes no warranties, express or implied, in or relating to this document or any information in it, including, without limitation, that this document, or any information in it, is error-free or meets any conditions of merchantability or fitness for a particular purpose. This document may not be reproduced or transmitted in any form or by any means without our prior written permission.

The material provided is for informational purposes only, and should not be relied on in making a purchasing decision. The information is not a commitment, promise or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for our products remains at our sole discretion.

During the course of this presentation TIBCO or its representatives may make forward-looking statements regarding future events, TIBCO’s future results or our future financial performance. These statements are based on management’s current expectations. Although we believe that the expectations reflected in the forward-looking statements contained in this presentation are reasonable, these expectations or any of the forward-looking statements could prove to be incorrect and actual results or financial performance could differ materially from those stated herein. TIBCO does not undertake to update any forward-looking statement that may be made from time to time or on its behalf.

DISCLAIMER

3

Spotfire X with the A(X) Experience

© Copyright 2000-2018 TIBCO Software Inc.

AugmentedSearch & AI-Powered Insights

Start in seconds, get instant insights

AutomatedAutomagical Dataflows

Author & audit with automaticallyrecorded dataflow steps

AgileReimagined User Interface

Agile exploration made even easier

AcceleratedReal-time Data Streams

Real-time awareness and action

Analytics,Accelerated.

4

Spotfire X with the A(X) Experience


AugmentedSearch & AI-Powered Insights

Start in seconds, get instant insights

AutomatedAutomagical Dataflows

Author & audit with automaticallyrecorded dataflow steps

AgileReimagined User Interface

Agile exploration made even easier

AcceleratedReal-time Data Streams

Real-time awareness and action

Analytics,Accelerated.

• Artificial Intelligence and Machine Learning

• AI Applications Using TIBCO Spotfire®

• TIBCO Spotfire® and the AI-Engine inside

• Demonstration

• To the Future

Agenda


Decision ManagementAI rules systems for decision support and automation

AI-optimized hardware/softwareGPU and appliances

Machine / Deep Learning PlatformsAlgos, APIs, dev/training toolkits; Insight Platforms

Virtual AgentsChatbots and interfaces

Natural Language Generation (NLG)Reports, BI Insights, customer service

Text Analytics / Natural Language Processing (NLP) Sentiment, intent analysis

Speech RecognitionIVR, mobile

Biometrics & Robotic Automation Virtual reality / automation

Artificial Intelligence – Hot Areas: Forrester


7

Supervised – Model known problems: y=f(X)

• Build a model that predicts an observed condition

• What factors are driving failures?

• Issue: needs lots of labeled data

Unsupervised – Explore observed data: X only

• Understand structure, detect anomalies

• Are there new patterns or failure modes emerging?

Machine Learning finds predictive models in data without being told where to look

PCA Autoencoder Single Class SVM

Machine Learning

DimensionReduction/

Anomaly Detection


8

AI Applications with Spotfire

Critical business moments occur in every facet of enterprise operations.

They drive competitive differentiation, customer satisfaction and business success.

Inventory Management

Customer Engagement

Recommender Systems

Anomaly Detection

Patient Diagnosis and Treatment

Fraud DetectionPrice

Optimization

Intrusion Detection

AI Success Stories – Critical Business Moments

Risk Management


• Supervised & unsupervised learning combine to recognize known risk patterns and identify anomalies

• Models deployed and applied to new incoming data

• Suspicious cases are case managedand then absorbed into the data for future model updates

AI on-demand / Continuous Learning

Augments People - Create / Operate Systems that actively manage Risk

AI Application Example: Risk Management


TIBCO Runtime for R(TERR™)

TIBCO Data Science™ Python

Spotfire Data FunctionSpotfire Expression

Spotfire Data Functions - Enable AI / ML Applications


12

Spotfire X

AI Engine

Direct manipulationSearchAutomatic Insights Data Workflow

AI Suggestions NLQ Auto Lineage Visual Analytics

Spotfire X: Multi-modal UX


14

Spotfire X: AI SuggestionsAutomated, Augmented insight discovery & display

Variable Relationships Algorithm• User selects target variable

• AI algorithm discovers variable relationships to target

• Spotfire displays in order of strength - best practices graphics

• 4 clicks to brush-linked dashboard

Users and Data• Broad user persona

• Scalable – including wide data

• Flexible – iterative, rapid configuration


Demonstration

AI Suggestions: Under the Hood

Input Data Table

Attributes à Features (p)

Row

s =

Rec

ords

(n)

Attributes• The data’s DNA: they make a record what it is• Engineered into Features: foundation for Machine Learning

Spatial and Date/Timecomplex hierarchies

Categoricalno scale, frequencysub-types: binary, ordered

Attribute types require different prep for Machine Learning

Numeric scale measure


Data Preparation & Feature Engineering

MetadataDiscovery

Data Cleaning

Feature Engineering

Remove uninformative records, attributes

Transform and add features

Metadata: Data about the attributes

Discover: the shape and spread of the attributes

Data manipulation - explain and empower, not dictate

• Data is minimally cleaned: just enough to be used with our algorithms + optimize pattern detection• Data bins, category collapse, time and space spans to optimal granularity, …

• Every feature update is completely transient: the user sees the real original data


Target and Feature Relationships

Attributes à Features

Row

s =

Rec

ords

When two features vary together in a non-random way, they have a relationship.

Linear Non Linear

• Linear relationships are easier to detect

• Spotfire’s AI Engine discovers linear and non-linear relationships

Target variable is of particular interest e.g. a KPI.

For a given Target, we identify features that are most related. Sometimes feature interactions show strong relationships to the Target.

We rank features and interactions by variable importance test statistics.


Select a Target

UX Workflow: AI Suggestions and beyond

See Graphs

Data preparation

Feature relationships

Ranking

+MORE LIKE THIS shows extra graphs with the same features

Adds graph to the pageAnalyst Business User

Tell me about “Target”

Continue on the journey: adorn, manipulate, explore, drill


AI & Variable Relationships: Summary

AI + People = Smarter Together

• AI-engine finds potentially interesting insightsà Less work to explore data (esp wide data)

• Human determines actual useful insightsà Curate final analysis, visuals, dashboard

AI Engine

• Based on well-established Data Science algorithms

• Customized for robustness and scale• Background processing for performance &

scalability


22

AI-Engine Design - Lessons Learned

Performance & Scalability• Require AI-engine results that appear in UX millisecond response times

Simplicity• Minimal algorithm configuration/tuning requirements • Emphasis on guiding user to next stage of analysis

Extensibility• Desire for tailored algorithms for domain specific vs. general use

Cross-functional Collaboration• Tight communication combining domain knowledge of data scientists with

software engineering


23

To the Future

Data Preparation• Simple & complex missing data patterns• Unusual string patterns• Unusual data distributions

Adornments• Patterns & Outliers

Time Series• Anomalies, patterns & relationships

Data Science on Data Science• Personal + ‘Crowd’ Recommendations


24

TIBCO Connected Intelligence Platform


Data Visualization

Data Science

Data Management

Integration andAPI Management

Messaging andEvents Processing

Digital ProcessAutomation

25

Spotfire 10 AI-Engine Team

• Dan Rope, Andrew Berridge, Gaia Paolini, Siva Ramalingam

• Erik Brandin, Kristin Edvardsen, Niklas Amberntsson

Acknowledgements


26

Questions & Contact


Thank you!Michael O’ConnellChief Analytics OfficerTIBCO [email protected]@MichOConnell

TIBCO Communitycommunity.tibco.com

TIBCO Exchangecommunity.tibco.com/exchange

Spotfire Trialspotfire.tibco.com/trial

Documents

2018-09-25 AI in Spotfire X Inside and Out MOC Final1 · AI in Spotfire X: Inside and Out This document (including, without limitation, any product roadmap or statement of direction