MongoDB and RDBMS: Using Polyglot Persistence at Equifax

1

MongoDB and RDBMS: Using Polyglot Persistence at Equifax

MongoDB Evenings Atlanta

September 24, 2015Mike Lawrence

2

“I specialize in business development utilizing a strong background in data science and architecture to improve business

Go-To-Market strategies and operation. I enjoy leveraging data to spot industry trends, make predictive decisions about

future growth areas, and improve context capture for data sets. I am also a caffeine-aholic, so please feel free to say hello

to me next time you’re at Starbucks. “

Mike LawrenceAssociate, Pariveda Solutions

@theMrLawrence

3

You will develop a strong understanding of the polyglot persistence usecase

Three Key Takeaways

1. Breaking traditional data storage patterns enabled Equifax to develop data persistence and access patterns for agility

2. The key drivers to implement MongoDB and the benefits to the business and consumer experience

3. Leveraging the strengths of MongoDB and RDBMS provides a versatile data solution that increase the lifetime value of consumer relationships and improve customer experience.

4

From business overview to solution architecture

A Look at our Presentation Agenda

Equifax PSOLA quick overview of the Equifax

Personal Solutions business unit

Data Access PatternsDiving into the data persistence and access patterns by the application

5

Understanding the DataExplore the different types of data

and understanding its use

Document StorageUse case, adoption, advantages of a document storage solution

Cost Savings with MongoDBA document storage solution

provided a reduction of overall storage costs

Relational StorageNot all data gets persisted into MongoDB, some data remains highly relational

6

Let’s begin!

Solution ArchitectureA quick glance into the polyglot

solution architecture between app and persistence

Q & ATime for all of your questions and comments!

7

Fueling New Product Innovation (NPI)

Equifax Personal Solutions

Consumer Impact

Personal Solutions continued to increase the lifetime value of its consumer relationships by improving the customer experience and

introducing new, high-value products

Equifax Personal Solutions, which contributes 10% of the overall Equifax Revenue, supplies consumers with information to help them understand their credit and protect their identity. In 2014 they launched a strategic transformation to ensure long-term, sustainable

growth in the face of a changing market environment.

8

The first step to the future

Equifax PSOL Strategic Transformation

Equifax is re-engineering the consumer application platform to better reach the digital consumer

A core principle of this strategic project is to introduce new technologies to Equifax that further expand their ability to execute on business objectives at lower operating costs and improve overall system performance.

9

Breaking Traditional Data Storage PatternsAdopting new approaches to data persistence and data access for agility

MapReduce

Data Processing for ComplexBI and Reporting

Streaming

Realtime processing and fulfillment

DocumentTransactional

Document Storage for cohesive and large transactional data

Relational Transactional

Relational storage for highly structured transactional data

DocumentArchival

Document Storage for Archival Solutions

Data Access Patterns

MapReduce

Data Processing for ComplexBI and Reporting

Streaming

Realtime processing and fulfillment

DocumentTransactional

Document Storage for cohesive and large transactional data

Relational Transactional

Relational storage for highly structured transactional data

DocumentArchival

Document Storage for Archival Solutions

Data Access Patterns

12

The types of data are wide ranging, but centered around the consumer

Application Data is Consumer Centric

Consumer InformationBasic information about the consumer must be persisted

to track identity

User AuthenticationRole management, user, and

customer verification are required for privileges

Order ManagementOrders are tracked through placement to completion

Product CatalogsAvailable products, offers, and cross sell are managed

through the database

ConfigurationsApplication configurations are stored for light payload

Audit LoggingAll activities must be tracked,

audited, and persisted

Digital ProductsCredit products are large documents of data to be supplied to a customer

Alert ProcessingAlerts are a form of product

that are persisted and supplied to consumers

13

Relational


to track identity












14


to track identity












Document

15

Document StorageAdoption, advantages and document types

16

As Equifax grows their consumer base, new data storage technologies were explored to keep with increase demand

Performance and costare two key drivers of success

Developing a new platform offered an opportunity to explore

different technologies solve new challenges

17

As development moved forward, the need document storage became clearer

Building a Case forDocument Storage

Lightweight Searchable StorageStorage of documents in RDBMS is bulky, slow retrieval, and

difficult to search

Data VolumeHigh volume of data creation and retrieval requires scalability

Last Mile DeliveryPerformance drives realtime rendering of credit reports, invoices, and other large documents

Realtime PerformanceLarge volumes of data are analyzed realtime by the business

18

The path that led Equifax from concept to adopting MongoDB

Choosing MongoDB as the Solution

Adoption of MongoDB is driven by retrieval, scalability, and cost

• MongoDB offers flexible storage, easy scalability, andhigh-performance searching and document retrieval

• High performance searching and retrieval allowsEquifax to render credit reports instantly

• MongoDB is a low-cost solution compared to RDBMSstorage of documents

• Scalability of MongoDB meets future growth needs ofEquifax as their data continues to grow exponentially

• Independent searching outside of RDBMS

High Volume Data Stored in RDBMS CLOB/BLOB

Fast Retention and Retrieval of Large Data

NoSQL Data Store

MongoDB

19

Determining where to persist data is decided from a few key rules

Rules for Data Persisted in MongoDB

Cohesive Unstructured Unknown Metadata High Volatility

Highly cohesive data or information that cannot be

broken down

Unstructured documents with few or no standards

New products and developments may require

different metadata

Documents susceptible to frequent schema changes

20







Documents Stored in MongoDB

21

What cost savings have we experienced with MongoDB?

22

A dramatic reduction in cost over relational storage

MongoDB Helped Increase Bottom Line

MongoDB Storage Cost per GB

Reduction of storage costs have been a major driving

force behind the implementation of MongoDB to

supplement the relational database.$2/gb

MongoDB$/gb

RDBMS$/gb

Storage costs were reduced

400% from $8/gb using

RDBMS to $2/gb using

MongoDB

400%Cost savings per GB

$ $ $ $ $ $ $ $

23

Relational StorageLeveraging existing investments

24

Determining where to persist data is decided from a few key rules

Rules for Data Persisted in RDBMS

Loosely Coupled Highly Structured Highly Related Low Volatility

Data that withstands the breakdown into smaller pieces

Hierarchical or other defined structures

Data extends out to many associations

Relational Data tends to go through few and minor

changes over time

25

Relational


to track identity






26

Polyglot Persistence Solution ArchitectureMongoDB and RDBMS Work Synchronously

27

Pers

iste

nce

Tra

nsa

ctio

n

Polyglot Solution Architecture with Referential Integrity

Persistence Service

Data Entry

Transaction

MongoDB RDBMS

Poly

glo

t Pe

rsis

ten

ce A

rch

itec

ture

Virtual Relational Integrity

28

Polyglot Persistence Provides aVersatile Data Solution

MongoDB and RDBMS working in harmony

29

Leveraging Both Storage PlatformsEnables Scalability, Performance, and Agility

MongoDB and RDBMS each have their place, using both together increases flexibility and growth

RDBMSHighly Relational or Structured

Transactional Data

Loose Cohesion

Low VolatilityRDBMS MongoDB

MongoDBUnstructured

Unknown Metadata

Tight Cohesion

High Volatility

Large Data

30

Branched Consumer Data with Document Leaf NodesThe complete consumer is much like a tree and its leaves

Credit Files

Audit Logs Alerts

Configurations

The polyglot persistence architecture leverages the

strengths of both storage technologies. The natural

structure of the consumer and the product catalog dictate

relational, while the products to be fulfilled are highly

cohesive documents.

The consumer tree and the document nodes.

A tree and its leaves

31

WHAT WHY WHERE WHEN WHO HOW

Q&AMongoDB and RDBMS: Using Polyglot Persistence at Equifax

Technology

MongoDB and RDBMS: Using Polyglot Persistence at Equifax