Testament - a blueprint for hyper-personalized services at web scale

Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0

introducing

the Testament Architecture

● An architecture for hyper-personalised services in an internet of things that you can use.

Robert Gibbon – be.linkedin.com/in/robertgibbonThe content of these slides is my own original thinking; or derived from information freely available in the public domain.


http://be.linkedin.com/in/robertgibbon


who am I?

● I am an independent information technology services consultant

● 15 years in the business

● 5 years doing stuff with big data

● And some more years on web-scale site management duty



why did I make this deck?

I made this deck because

● I had a collection of ideas that were buzzing around in my head and I wanted to get them out onto paper.

● Neither the concepts and technologies, nor the way they are combined and presented in this architectural paper are related to any of my past or present clients.

● But...if you try to implement a Testament architecture and it doesn't work out for you, don't blame me!



contents

This slide set goes over

● The operational context of the Testament architecture

● The challenges to service delivery that the Testament approach prioritises

● A flythrough of the major features and technologies in a Testament implementation

● Some closing thoughts



who stands to benefit from Testament?

Testament is a high level architecture blueprint. It is oriented to multifaceted consumer services that require

● global reach

● millions of users

● multiple devices per user

● always available

● long data retention times

● processing very big datasets

Testament proposes a set of building blocks on which to found certain aspects of such a service.



what challenges does Testament try to address?

Testament prioritises these aspects of successful hyper-personalized service delivery at web scale.

● serving hyper-personalised data to millions+ concurrent users

● serving and reconciling data with split-brained services and offline-often devices

● serving “single record of truth” datums that must always be kept up to the minute

● cost-critical long term data storage and access for very big data



front end: conceptually

Content Distribution Network

Global Service Center 1

User X Database Branches User Y Database Branches

User X Managed Devices

CHICAGOAMSTERDAM

TOKYO

Global Service Center 2 Global Service Center n

Communal Edge Service Nodes User Y Managed Devices



front end: logically

Distributed Filesystem

VCS, DDB, CEP

Offline-often Devices

CHICAGOAMSTERDAM

TOKYO

VCS, DDB, CEP VCS, DDB, CEP

Hosted Application Servers Offline-often Devices


Standalone dbs Standalone dbs


front end: exemplar physical model

XtreemFS

Git, Hbase, Storm

Sunglasses, Phones, Wristbands

CHICAGOAMSTERDAM

TOKYO

Git, Hbase, Storm Git, Hbase, Storm

Games, eShops, Content on-demand, Social TVs, Hi-Fis, Fridges


SQLite files SQLite files


front end: summary of the bits

Testimonies

● Are a user, session, and device bounded collection of events and facts

● eg. “A testimony of latlongs, browsing events, DNS lookups and app usage activities from Robert's phone for last Friday”

● These testimonies are built to help better understand the user and consequently better serve the user, over time

Global Service Centers

● Handle asynchronously submitted testimonies

● Pull new testimonies from the CDN

● Automerge and tag them, eg. “Merge this testimony onto the <Robert's phone> branch and tag it <last Friday>”

● Precompute ahead of pull requests from Edge Service Nodes for updated User Database Branches

● Run “single-version-of-the-truth” database services for those datums that must always be up to the minute, such as authentication credentials, avatars, screen names etc.





● Common service backplane

● Write-locally, read-globally

● Ship User Database Branches between Global Service Centers and Edge Service Nodes

● Ship Testimonies between Edge Service Nodes and Global Service Centers

User Database Branches

● Provide a self-contained individual user database to enable hyper-personalized service delivery

● eg. attitude/trend/interest/demographic classifiers; motivators, intents and triggers; recencies; rising and decaying trends

● Form a container for Testimonies to be backshipped via the Content Distribution Network




Communal Edge Service Nodes and Managed Devices

● Deliver actual application services to the user

● eg. Games, grocery shopping, grocery storage, streaming radio, TV on demand, household automation, remote power/HVAC management, social etc.

● Are two-way, in that they build and backship Testimonies, to help better understand the user and consequently better serve the user, over time



back end: conceptually


Global Service Center 1

CHICAGOTOKYO

AMSTERDAM

VANCOUVER

HEBRIDES

ATLANTA

Global Service Center 2 Global Service Center n

Nearline Vault 1 Nearline Vault 2 Nearline Vault n



back end: logically

Distributed Filesystem

Compute Grid 1

CHICAGOTOKYO

AMSTERDAM

VANCOUVER

HEBRIDES

ATLANTA

Compute Grid 2 Compute Grid n

Massive Array of Idle Devices Massive Array of Idle Devices Massive Array of Idle Devices



back end: exemplar physical model

XtreemFS

Spark on Mesos

CHICAGOTOKYO

AMSTERDAM

VANCOUVER

HEBRIDES

ATLANTA

Spark on Mesos Spark on Mesos

Depreciated Commodity Systems Depreciated Commodity Systems Depreciated Commodity Systems



backend: summary of the bits

Global Service Centers

● Deliver the usual background batch processing services to build, refine and update apt and timely user insights from available Testimonies, so that front-end applications can better serve the user

● Identify Testimonies that the Global Service Centers cannot use at this moment and that are therefore eligible to be archived to the Nearline Vault

● Eg. Analysis technology is not yet sufficiently mature, and/or Testimony content has decayed in relevance


● Acts as a publish locally, read remotely conduit from Global Service Centers to Nearline Vaults



backend: summary of the bits

Nearline Vaults

● Load old Testimonies onto a storage device on a Depreciated Server, spin down the storage device, and power off the Depreciated Server for long term storage

● Power up the Depreciated Server and spin up the required storage device on demand, for retrieval of the old Testimonies in the future.

Depreciated Servers

● Old Servers that have depreciated, configured with high storage device density, often racked and packed in self-contained DC freight containers.



closing thoughts

Testament only considers a small subset of architectural facets; its really only a teaser.

There is a spectrum of architectural facets to desigining a successful hyper-personalised, web-scale service that shall be considered, and that are not considered by Testament.

For example, User Database Branches and Testimonies must be protected and able to defend against unauthorised access, such as from competing service providers or from malicious users.

Also, the personally identifiable link between Testimonies and the user to which they relate and the legal and ethical rights of users to sever that link, is another facet that these slides have not considered.

Importantly, “no man is an island”, and handling Testimonies of the relationships between users has not been discussed in this slide deck.



slide deck license

Copyright 2014 Robert Gibbon

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.


http://www.apache.org/licenses/LICENSE-2.0

Technology

Testament - a blueprint for hyper-personalized services at web scale