Big Retail Goes Reactive at Walmart

REACTIVE RETAIL BIG RETAIL GOES REACTIVE AT WALMART

Reactive Retail

Big Retail Goes Reactive at Walmart NOVEMBER 2014


Introduction Walmart.ca re-architecture Move to Functional Programming Technical Tips & Features Experiences and Impressions Outlook Q&A

Agenda


About Nurun

12

Atlanta

Nancy

Shanghai

Quebec Turin

Montreal

Paris

Milan

Toronto

San Francisco

Chicoutimi

Madrid

1,200+ Employees

14 Years In Business

Retail

Automotive

Financial Services

Media & Entertainment

Fashion & Beauty

AREAS OF EXPERTISE

Utilities

Government

Telecommunications

Travel & Hospitality

Health & Pharma

3

We are a global design & technology consultancy with 12 offices worldwide.


Introduction


Introduction

TYPICAL LARGE ECOMMERCE ENVIRONMENTS

E-Commerce Platform

RDBMS

JSPs

Product content management

CDN


Introduction

TYPICAL LARGE ECOMMERCE ENVIRONMENTS

E-Commerce Platform Search &

merchandizing

RDBMS

JSPs


CDN


Introduction

INCUMBENT PLATFORM

•  ATG, Oracle •  Fairly monolithic

•  Rapidly evolving merchandising needs


Introduction

INCUMBENT PLATFORM




Introduction

INCUMBENT PLATFORM



•  Scalability issues


Retail priorities

EFFICIENT & SCALABLE Need to handle large volumes RESILIENT Downtime is expensive FLEXIBLE & MAINTAINABLE Business requirements change


Need to go Reactive to be …

RESPONSIVE Customer experience, brand promise RESILIENT Handle failure at multiple levels ELASTIC Handle traffic spikes (without breaking the bank) MESSAGE-DRIVEN Enable all of the above


Walmart.ca Re-architecture


Walmart.ca re-architecture

ATG

Oracle


JSPs



ATG

Oracle

REST API

Endeca search & merchandizing

REST API


JSPs



ATG

Oracle


REST API

Akka, Redis


JSPs



ATG

Oracle


REST API

Play

Akka, Redis


JSON

JSPs



ATG

Oracle

REST API


REST API

Play

Akka, Redis


JSON JSON



No. 1 +20%

+98%

Canada’s first responsively designed e-commerce site for a big box retailer

Increase in conversion in the first 4 weeks after launch

Increase in mobile orders

-36%

Decrease in average page load time

Awarded BEST E-COMMERCE SITE in large retailers by Retail Council of Canada



No. 1 +20%

+98%

Canada’s first responsively designed e-commerce site for a big box retailer

Increase in conversion in the first 4 weeks after launch

Increase in mobile orders

-36%

Decrease in average page load time

NO SYSTEM OUTAGES!

Awarded BEST E-COMMERCE SITE in large retailers by Retail Council of Canada


Move to Functional Programming



● Majority of the team are developers with a Java background

● Play provides an easy entry path to Scala / Reactive / Functional:

§  Based on familiar MVC patterns

§  Can start with Java style, mutation, blocking

§  Incrementally transition to Scala style, immutable, non-blocking



1. No nulls (use Option)

2. No vars, mutable state*

3. Use higher-order functions (map, flatMap, filter, for-yield, etc.)

4. Business logic should be side-effect free (testability!)

5. Create and use strong types

*unless well justified (e.g. much clearer), and as small a scope as possible



●  Code should be a clear and concise as possible

●  Types should be express intent and fit tightly


Move to Functional Programming: Strong Types

sealed trait Language case object English extends Language

case object French extends Language def formatPrice(price: Price, lang: Language) = lang match {

case English => … case French => …

}

implicit val binder = new PathBindable[Language] {

… }


Technical Tips & Features


Tips & Features: Caching Futures (on-heap)

● Several concurrent requests for same data

● Whoever requests first initiates the Future (Promise)

● No locking required (narrow window between first request and when

Future is cached)


Tips & Features: Handling JSON

Play Application Makes Heavy Use of JSON

•  Consumes JSON view advice from Endeca

•  Proxy & augment JSON data for web service calls


Tips & Features: Play JSON macro

Consuming JSON (e.g. Endeca cartridges) case class Product(id: Long, name: String, …)

object Product {

implicit val formats = Json.format[Product] }

Json.toJson(product)

productJson.asOpt[Product]


Tips & Features: ‘Almost Default’ Reads/Writes

OPEN CHALLENGE

Sometime we want an ‘almost default’ Reads/Writes

•  e.g. use a default

•  Read/write a few fields in a particular way

•  e.g. format two dates in different ways

•  (Could Shapeless case class generics macro help?)


Tips & Features: Proxying and Augmenting JSON

Proxying JSON (e.g. most web services)

•  Might want to augment a JSON response

•  Don’t need to care about knowing the entire structure

•  Play JSON transformers

•  play-json-zipper can be helpful (github: mandubian)


Tips & Features: Dependency Injection

CONTROLLER DEPENDENCIES (SERVICES)

•  Use structural types for DI at the controller level

•  Create a services module for each request

•  Want access to request header (logging)


Tips & Features: Dependency Injection

THREADLOCAL

•  Inject dependencies in Global.getControllerInstance

•  Initialize services module in Global.onRouteRequest, add to

ThreadLocal

•  Pick up this value in the getControllerInstance and provide it to our

controller


Tips & Features: Monadic Rendering

Page Model

•  Template

•  Zones

•  Cartridges


Rendering View Advice: Templates & Zones

The business user selects a template:

•  Determines the page layout

•  Each template has ~12 zones


Rendering View Advice: Cartridges

… adds cartridges to the zones:

•  ~100 cartridges

ProductImage

ProductDescription

ProductPurchase


Rendering View Advice: Whole Page

•  Endeca fills in data (product info)

•  Play renders the cartridge as HTML


Rendering View Advice: JSON to Visual Page •  Tmpl1

•  Header •  Car1 •  Car2

•  LeftNav •  Car3

•  MainContent •  Car4 •  Car5 •  Car6


Rendering View Advice: JSON to HTML •  Tmpl1

•  Header •  Car1 •  Car2



<body>! <div class=“header”>!! !…!

</div>! <div class=“leftNav”>!! !…!

</div>! !<div class=“main”>!! !…!

</div>!</body>!


Rendering View Advice: Auxiliary Information

•  Tmpl1 •  Header

•  Car1 •  Car2



<head>!

!

</head>!

<body>!

<div class=“header”>!

! !…!

</div>!

<div class=“leftNav”>!

! !…!

</div>!

!<div class=“main”>!

! !…!

</div>!

!

!

</body>!







<head>!

!

</head>!

<body>!


! !…!

</div>!


! !…!

</div>!


! !…!

</div>!

!

!

</body>!

CSS / JavaScript / RJS modules







<head>!

!

</head>!

<body>!


! !…!

</div>!


! !…!

</div>!


! !…!

</div>!

!

!

</body>!


JavaScript / RJS modules







<head>!

!

</head>!

<body>!


! !…!

</div>!


! !…!

</div>!


! !…!

</div>!

!

!

</body>!


JavaScript / RJS modules Cache control


Monadic Rendering: ValidatedBuilder Monad

ValidatedBuilder[X] •  “Parse/process” X •  ≅ Future[(X, Boolean, Resources)]

•  Error flag: • Determine if page should be cached

•  Front-end resources •  CSS • RequireJS modules •  JavaScript snippets (+ page location requirements)


Monadic Rendering: ValidatedBuilder Monad

ValidatedBuilder[X] •  map and flatMap handle combining and propagating validity and

front-end resources •  Code focuses on the essential task (JSON -> HTML) without clutter •  Error prone ‘plumbing’ is handled by the monad abstraction


Monadic Rendering: ValidatedBuilder Zone

val headerContent: VB[Seq[Cartridge]] = (viewAdvice \ “header”)

.as[Seq[JsValue]] // Seq[JsValue] .map(Cartridge.parse) // Seq[VB[Cartridge]]

.vbSeq // VB[Seq[Cartridge]]


Monadic Rendering: ValidatedBuilder Template

val template: VB[Template] = for { header <- headerContent : VB[Seq[Cartridge]]

leftNav <- leftNavContent …

templateResult <- CayenneTemplate(header, leftNav, …)

} yield templateResult


Monadic Rendering: ValidatedBuilder Page

for { content <- template.contents : Future[Template]

resources <- template.resources : Future[Resources] isCacheable <- template.isValid : Future[Boolean]

yield {

val result = template.render(resources) if (isCacheable)

result else

result.withHeaders(noCache: _*)

}


Tips & Features: Dynamic Configuration

•  Ops can make on the fly configuration changes

•  Enable/disable services and site features

•  Change endpoints


Tips & Features: Dynamic Configuration

•  Sends configuration delta to Play (JSON object)

•  Play verifies the change (initialize services module)

•  Each request sees a snapshot of the configuration

•  Uses standard Play HOCON API


Experience and Impressions


Experience and Impressions ● Never need to say “no” to a client due to technical limitations,

● Heavily concurrency / parallelism, with very little headaches,

● Aggressive refactorings are surprisingly easy (thank you strong types).


Outlook


Outlook

NEW BREED OF PRODUCTS Reactive core


Outlook

NEW BREED OF PRODUCTS Reactive core Middleware


Outlook

NEW BREED OF PRODUCTS Reactive core Middleware Omnichannel CE


Outlook

NEW BREED OF PRODUCTS Reactive core Middleware Omnichannel CE Streaming big data Machine intelligence


Outlook

NEW BREED OF PRODUCTS Reactive core Middleware Omnichannel CE Streaming big data Machine intelligence CMS


Outlook

NEW BREED OF PRODUCTS Reactive core Middleware Omnichannel CE Streaming big data Machine intelligence CMS BAAS REsponsiveSS


Outlook

NEW BREED OF PRODUCTS Reactive core Middleware Omnichannel CE Streaming big data Machine intelligence CMS BAAS REsponsiveSS In store tech, IoT…


Q & A

Ankur Mathur @anchormath Dana Harrington @danagharrington https://github.com/dana-harrington

Technology

Big Retail Goes Reactive at Walmart