Click here to load reader
Upload
robert-gibbon
View
210
Download
0
Embed Size (px)
DESCRIPTION
This is an architectural design pattern I put together to address a real world, but fairly generic, set of requirements that seem to come up in discussion time and again. It is by no means meant to suggest a complete panacea formula...just a few concept sketches really.
Citation preview
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
introducing
the Testament Architecture
● An architecture for hyper-personalised services in an internet of things that you can use.
Robert Gibbon – be.linkedin.com/in/robertgibbonThe content of these slides is my own original thinking; or derived from information freely available in the public domain.
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
who am I?
● I am an independent information technology services consultant
● 15 years in the business
● 5 years doing stuff with big data
● And some more years on web-scale site management duty
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
why did I make this deck?
I made this deck because
● I had a collection of ideas that were buzzing around in my head and I wanted to get them out onto paper.
● Neither the concepts and technologies, nor the way they are combined and presented in this architectural paper are related to any of my past or present clients.
● But...if you try to implement a Testament architecture and it doesn't work out for you, don't blame me!
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
contents
This slide set goes over
● The operational context of the Testament architecture
● The challenges to service delivery that the Testament approach prioritises
● A flythrough of the major features and technologies in a Testament implementation
● Some closing thoughts
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
who stands to benefit from Testament?
Testament is a high level architecture blueprint. It is oriented to multifaceted consumer services that require
● global reach
● millions of users
● multiple devices per user
● always available
● long data retention times
● processing very big datasets
Testament proposes a set of building blocks on which to found certain aspects of such a service.
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
what challenges does Testament try to address?
Testament prioritises these aspects of successful hyper-personalized service delivery at web scale.
● serving hyper-personalised data to millions+ concurrent users
● serving and reconciling data with split-brained services and offline-often devices
● serving “single record of truth” datums that must always be kept up to the minute
● cost-critical long term data storage and access for very big data
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
front end: conceptually
Content Distribution Network
Global Service Center 1
User X Database Branches User Y Database Branches
User X Managed Devices
CHICAGOAMSTERDAM
TOKYO
Global Service Center 2 Global Service Center n
Communal Edge Service Nodes User Y Managed Devices
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
front end: logically
Distributed Filesystem
VCS, DDB, CEP
Offline-often Devices
CHICAGOAMSTERDAM
TOKYO
VCS, DDB, CEP VCS, DDB, CEP
Hosted Application Servers Offline-often Devices
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Standalone dbs Standalone dbs
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
front end: exemplar physical model
XtreemFS
Git, Hbase, Storm
Sunglasses, Phones, Wristbands
CHICAGOAMSTERDAM
TOKYO
Git, Hbase, Storm Git, Hbase, Storm
Games, eShops, Content on-demand, Social TVs, Hi-Fis, Fridges
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
SQLite files SQLite files
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
front end: summary of the bits
Testimonies
● Are a user, session, and device bounded collection of events and facts
● eg. “A testimony of latlongs, browsing events, DNS lookups and app usage activities from Robert's phone for last Friday”
● These testimonies are built to help better understand the user and consequently better serve the user, over time
Global Service Centers
● Handle asynchronously submitted testimonies
● Pull new testimonies from the CDN
● Automerge and tag them, eg. “Merge this testimony onto the <Robert's phone> branch and tag it <last Friday>”
● Precompute ahead of pull requests from Edge Service Nodes for updated User Database Branches
● Run “single-version-of-the-truth” database services for those datums that must always be up to the minute, such as authentication credentials, avatars, screen names etc.
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
front end: summary of the bits
Content Distribution Network
● Common service backplane
● Write-locally, read-globally
● Ship User Database Branches between Global Service Centers and Edge Service Nodes
● Ship Testimonies between Edge Service Nodes and Global Service Centers
User Database Branches
● Provide a self-contained individual user database to enable hyper-personalized service delivery
● eg. attitude/trend/interest/demographic classifiers; motivators, intents and triggers; recencies; rising and decaying trends
● Form a container for Testimonies to be backshipped via the Content Distribution Network
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
front end: summary of the bits
Communal Edge Service Nodes and Managed Devices
● Deliver actual application services to the user
● eg. Games, grocery shopping, grocery storage, streaming radio, TV on demand, household automation, remote power/HVAC management, social etc.
● Are two-way, in that they build and backship Testimonies, to help better understand the user and consequently better serve the user, over time
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
back end: conceptually
Content Distribution Network
Global Service Center 1
CHICAGOTOKYO
AMSTERDAM
VANCOUVER
HEBRIDES
ATLANTA
Global Service Center 2 Global Service Center n
Nearline Vault 1 Nearline Vault 2 Nearline Vault n
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
back end: logically
Distributed Filesystem
Compute Grid 1
CHICAGOTOKYO
AMSTERDAM
VANCOUVER
HEBRIDES
ATLANTA
Compute Grid 2 Compute Grid n
Massive Array of Idle Devices Massive Array of Idle Devices Massive Array of Idle Devices
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
back end: exemplar physical model
XtreemFS
Spark on Mesos
CHICAGOTOKYO
AMSTERDAM
VANCOUVER
HEBRIDES
ATLANTA
Spark on Mesos Spark on Mesos
Depreciated Commodity Systems Depreciated Commodity Systems Depreciated Commodity Systems
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
backend: summary of the bits
Global Service Centers
● Deliver the usual background batch processing services to build, refine and update apt and timely user insights from available Testimonies, so that front-end applications can better serve the user
● Identify Testimonies that the Global Service Centers cannot use at this moment and that are therefore eligible to be archived to the Nearline Vault
● Eg. Analysis technology is not yet sufficiently mature, and/or Testimony content has decayed in relevance
Content Distribution Network
● Acts as a publish locally, read remotely conduit from Global Service Centers to Nearline Vaults
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
backend: summary of the bits
Nearline Vaults
● Load old Testimonies onto a storage device on a Depreciated Server, spin down the storage device, and power off the Depreciated Server for long term storage
● Power up the Depreciated Server and spin up the required storage device on demand, for retrieval of the old Testimonies in the future.
Depreciated Servers
● Old Servers that have depreciated, configured with high storage device density, often racked and packed in self-contained DC freight containers.
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
closing thoughts
Testament only considers a small subset of architectural facets; its really only a teaser.
There is a spectrum of architectural facets to desigining a successful hyper-personalised, web-scale service that shall be considered, and that are not considered by Testament.
For example, User Database Branches and Testimonies must be protected and able to defend against unauthorised access, such as from competing service providers or from malicious users.
Also, the personally identifiable link between Testimonies and the user to which they relate and the legal and ethical rights of users to sever that link, is another facet that these slides have not considered.
Importantly, “no man is an island”, and handling Testimonies of the relationships between users has not been discussed in this slide deck.
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
slide deck license
Copyright 2014 Robert Gibbon
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0