Optimizing the Data Supply Chain for Data Science

Optimizing theData Supply Chainfor Data Science

61 Broadway Suite 1105 New York, NY 10006

[email protected] http://www.vital.ai

Marc HadfieldCEO, Vital A.I.

mailto:[email protected]

http://www.vital.ai

about: vital ai



Software Applications:Artificial Intelligence, Machine Learning, Data Science.

Software Vendor & Consulting Services


http://www.vital.ai

agenda



• Data Models • How A.I., Data Science, & Data Governance relate • Data Supply Chain & the Data Product • Problem: the “Telephone Game” across the DSC • Architecture Transition from Data Warehouse to DSC • Data Models and DSC; a Framework for Solutions • Examples • Collaboration & Visualization

note: general methodology, with some specific examples from Vital AI implementations.


http://www.vital.ai

takeaways:



• The Data Supply Chain is a supply chain to deliver Data Products

• Data Models can capture the implicit meaning of data (and that is the goal!)

• Data Models can help negotiate the implicit differences across the DSC

• Data Models offer a means to collaborate on data standards (meaning) across the DSC partners


http://www.vital.ai

about data models:



Semantic Models


http://www.vital.ai

big data:



volume, velocity, variety, veracity

variety: data models“Product”: different meaning in Manufacturing vs Retail context

Healthcare, same entity: “Patient”, “InsuredPerson”, “BillableEntity”


http://www.vital.ai

example:



Class: PersonProperty: birthday

Standardized Unique Global Identifier (URI) data type: date relationship with property: age allowed range of values (can’t be born in the future) typical (average/expected) value…(Birthdays in Wikipedia vs Customer Database)


http://www.vital.ai

about: vital ai tech



Vital AI Development Kit (VDK)VitalSigns — Data Modeling & Code Generation

VitalService — Common API for Databases, Machine Learning, Apache Spark, Data Transforms


http://www.vital.ai

about: vital ai tech



VitalServiceQuery

ExecutableQuery

Query Generator

Common Query API:Relational DB (SQL) Graph DB (Sparql) Key/Value Store NOSQL DBDocument DBApache SparkHive (Hadoop) Predictive Models (a query for an unknown value)

Goal: Build A.I. applications across variety of infrastructure with consistent API & Models.


http://www.vital.ai

example data:



Person:Recipient

Person:Sender Message

hasRecipient

hasSender


http://www.vital.ai

example “MetaQL” query:



GRAPH { value segments: ["mydata"] ARC { node_constraint { Message.class } constraint { "?person1 != ?person2" } ARC_AND { ARC { edge_constraint { Edge_hasSender.class } node_constraint {

Person.props().emailAddress.equalTo(“[email protected]") }

node_constraint { Person.class } node_provides { "person1 = URI" } } ARC { edge_constraint { Edge_hasRecipient.class } node_constraint { Person.class } node_provides { "person2 = URI" } } } } }

“Person” may have subtypes, like Student or Employee.


http://www.vital.ai

a.i. and data quality




http://www.vital.ai

data models & machine learning:



using the meaning of classes and properties, automatically generate predictive models.

predictive models features:birthday, zip code, …


http://www.vital.ai

data governance =defining the meaning of data = feature (pre)engineering



critical aspect of data science


http://www.vital.ai



Progression of Analytics:


http://www.vital.ai



where a.i. happens

Progression of Analytics:


http://www.vital.ai

Garbage In = Garbage Out



= Bad A.I.

data governance required for Good A.I.


http://www.vital.ai



one more point ondata governance…

think outside the box(data warehouse)


http://www.vital.ai

data governance: data in motion



vs.inside data warehouse

outside data warehouse


http://www.vital.ai

supply chain




http://www.vital.ai

supply chain:



product


http://www.vital.ai

data supply chain:



data product


http://www.vital.ai



Retail Recommendations… Shipping/Logistics Optimization… Compliance, Auditing, Security, Fraud Detection…

data product:


http://www.vital.ai

why data supply chain?



Partner DW Your DW

"No matter who you are, most of the smartest people work for someone else.” — Bill Joy.


http://www.vital.ai

why data supply chain?



Partner DW Your DW

"No matter who you are, most of the smartest people data works for someone else.” — Bill Joy. (revised)


http://www.vital.ai

data supply chain



Partner DW

Your DW

why not ETL?


http://www.vital.ai

Partner DW



Extract…



not quite as expected…



Transform…

a bit extreme…

Load…



a bit messy…

Clean…



a lot of manual effort…

… your imported data61 Broadway Suite 1105

New York, NY [email protected]

http://www.vital.ai

Your DW61 Broadway Suite 1105


http://www.vital.ai

Partner DW

Why?



what goes wrong?

telephone game…

You61 Broadway Suite 1105


http://www.vital.ai

Partner

Model “A”

Model “B”

Implicit Model

Resolution: Make explicit the implicit. Align Data Models.



Reason:Implicit assumptions in the data. ETL can’t see the forest for the trees. (or it’s very difficult with missing assumptions)


http://www.vital.ai

Example: Internet of Things



Predictive Analytics

“Nest for Office Buildings” Office Tower with Building Management System (BMS) containing 100,000 monitored points (temperature, energy usage of chiller, fan speed, etc.) with significant missing data, errors, and noise. Reconciliation of data to produce predictive models to minimize energy usage. Rules for data correctness.


http://www.vital.ai

Sensor Data Validation:



Source data had temperature values of “0” (zero) which meant either the temperature was 0 degrees or that the sensor had an error.Data Model “knows” that it’s rarely 0 degrees in July (far from the standard deviation), and that the temperature can be compared to weather data on a day in December for reasonableness. If Data Model also knows the maintenance schedule for the sensors, then it “knows” when to expect 0 error values and exclude them.

Missing Maintenance Assumptions. Fill in secondary (weather) data for validation.


http://www.vital.ai

how did we get here?61 Broadway Suite 1105


http://www.vital.ai

Architecture Review:a quick step back…

What is a Data Supply Chain architecture?


http://www.vital.ai

“traditional” data warehouse:



ETL within the organization.Data Governance across the organization.

DW


http://www.vital.ai

tech co. “agile” data warehouse:



storage

compute

HDFS

Spark

DataSetsJobs

Batch/StreamingBuild Predictive Models Realtime: Spark/Storm

hadoop cluster


http://www.vital.ai

enterprise: data lake



storage

compute

HDFS

Spark

X(save $)

“Data Swamp”


http://www.vital.ai

aside: Data Lake



better analogy: Scriptorium

library,manuscript copying, & book distribution.

but not as Pithy as “Lake”…


http://www.vital.ai

tech co. microservices (micro-SOA):



storage

compute

service

“Composed” App

external: social data, weather API

independent clusters,local data expertise

optimize development processes, scale up.


http://www.vital.ai

microservices example:



Amazon: product search uses 170 independent microservices

including services for predicting customer characteristics, getting product images, etc.

http://www.infoworld.com/article/2903144/application-development/how-to-succeed-with-microservices-architecture.html

Netflix similar architecture


http://www.vital.ai

http://www.infoworld.com/article/2903144/application-development/how-to-succeed-with-microservices-architecture.html

Data Supply Chain:



storage

compute

service

Data Product

“ETL”

Owner “A” Owner “B”

optimize development processes, scale up.

independent clusters,local data, ownership


http://www.vital.ai

Interaction Points:



Data Product

service

compute ETL



http://www.vital.ai

Data Lineage: Cloudera Navigator



…within a Data Warehouse

trace back jobs that produced every data field.


http://www.vital.ai

Data Supply Chain with Provenance:



include provenance data directly in imported dataset. use in rules to interpret the data.

entity-123 | hasSource | datasource-A entity-123 | name | “John Doe”

Data Warehouse B


http://www.vital.ai

Interaction Points: Data Models



Data Product

service

computeETL

Data Models: Gatekeepers & Transform



http://www.vital.ai

Data Supply Chain using Models:



storage

compute

service

Data Product

ETL

Owner “A” Owner “B”Model Server

Data Models: focus of data governance


http://www.vital.ai

Semantic Data Models:



Make explicit the meaning of data

Transformation and Validation Rules leverage the Model and Meaning.Such Rules may be packaged with the Model, and managed together.

Protect against implicit assumptions


http://www.vital.ai

Example: Financial Services



A B C

Service Provider

Reconciliation of Corporate Structure across 1,000’s of organizations. Compliance Rules barring communication between “researchers” and “traders”.Rules to infer if “Mary” is a “researcher” or “trader”.Conflicting concepts of “Branch-Office”, “Direct-Report”, etc. across the Globe.


http://www.vital.ai

Example: Hospital Group



A B C

Data Analytics

Reconciliation across Patient Records, Insurance, & Billing for Patient Predictive Analytics.Rules for identity: “same person”


http://www.vital.ai

Data Models:



OWL: Semantic Ontology Model (W3C Standard, Various Standards for Rules)

VitalSigns: Generate Codevalidation, transformation, …

VitalSigns: Versioning, Dependencies, Exchange, Storage, Change Management (Semantic “Diff”)


http://www.vital.ai

Example: Personally Identifiable Information



Data Governance determines that “Profession” and “ZipCode” cannot be used together. (Maybe a single “Dentist” in a small town…)

Within a single Data Warehouse we can bar these data elements from being combined. But:Microservice A provides value of “Profession” Microservice B provides value of “ZipCode” How to enforce that these two microservices cannot be combined?


http://www.vital.ai

Example: Personally Identifiable Information



Validation code enforcing data usage:

Person person123 = get_person_details(“entity-123”) // this call works: person123.profession = get-profession(person123)// this call blocks because of data model validation // person123 already has “profession” propertyperson123.zipcode = get-zipcode(person123)


http://www.vital.ai



GatekeepersExternally Managed.Active not Passive, more like “code”.Defining what should exist, not cataloguing what exists.Can decide when to be tolerant or strict.

Semantic Data Models:


http://www.vital.ai

Collaborative Conversations:



Infrastructure DevOps

Data Scientists

Business +Domain Experts

Developers SemanticModel


http://www.vital.ai

Collaborative Conversations:




SemanticModel


SemanticModel

Partner A Partner B

Model Alignment

What Concepts to combine, not what Tables to combine (that comes later).


http://www.vital.ai

Authoring Tool: OWL IDE Protege




http://www.vital.ai

Visualization: Semantic Data




http://www.vital.ai

Visualization: WebVOWL




http://www.vital.ai

in conclusion, takeaways:



• The Data Supply Chain is a supply chain to deliver Data Products

• Data Models can capture the implicit meaning of data (and that is the goal!)

• Data Models can help negotiate the implicit differences across the DSC

• Data Models offer a means to collaborate on data standards (meaning) across the DSC partners


http://www.vital.ai

Questions?



Marc HadfieldCEO, Vital [email protected]


http://www.vital.ai

mailto:[email protected]?subject=

Data & Analytics

Optimizing the Data Supply Chain for Data Science