BBC's GraphDB (formerly Owlim) AWS Cloud Migration

Moving OWLIMto Amazon Web Services

Moving OWLIMto Amazon Web Services

GraphDB

Introduction to LDP

● Stands for Linked Data Platform● Building APIs and Tools for semantic

publishing at the BBC● Team of 8-12 people (5-8 developers) ● Based in London (moving to Salford)

Introduction to LDP

● Tagging and surfacing content

Why move to AWS?

● Started with taking ownership of Owlim● BBC shift to move to AWS● Loss of key people● Freedom with responsibility

Current Architecture

Cloud Architecture

Keeping in sync

OpsWorks

● Part of AWS● Easily deploy things to EC2

OpsWorks

● Define infrastructure in a stack● The stack is comprised of layers● Each layer has a role (eg. master, slave)● Layers contain a number of instances

OpsWorks – Advantages

● Conceptually, fairly simple.● Entire stack defined as JSON.

○ Can be (and is) version controlled.

● Very easy to clone an entire stack.○ We use this to spin up ‘experimental’ environments.○ Next step: blue/green deployments?

OpsWorks – Disadvantages

● It can be a little slow.● Chef support can be tricky to debug.

● “Infrastructure as code”● Cookbook, made up of recipes.

○ Each recipe installs/configures a specific thing.

● Each layer has a set of recipes that define it.

Chef

include_recipe 'tomcat::default'

cookbook_file 'owlim-production.license' do path File.join(node[:owlim][:basedir], 'owlim.license') mode 0644 owner 'tomcat' group 'tomcat'end

execute 'create_default_cluster_repository' do command SystemRepository.create_repo 'default-cluster' not_if SystemRepository.repo_exists? 'default-cluster' action :nothingend

Lifecycleevents

Recipes for each event type

● A common and well-documented tool.○ And it’s Ruby, a skill we possess.

● Can re-use existing cookbooks.○ eg. Tomcat, Yum.

● Should be platform-agnostic.○ Can theoretically use the same Chef recipes against

any OS, if written well.

Chef – Advantages

● OpsWorks support is a little lacking.○ Using Berkshelf for dependencies is tricky.○ Many of the error messages are a bit cryptic.

Chef – Disadvantages

What next?

What next?

● Autoscaling○ Not supported by OpsWorks.○ For now: predict spikes, use time-based instances.

What next?

● Autoscaling○ Not supported by OpsWorks.○ For now: predict spikes, use time-based instances.

● Backups○ We still do these from our DCs.

What next?

● Autoscaling○ Not supported by OpsWorks.○ For now: predict spikes, use time-based instances.

● Backups○ We still do these from our DCs.

● Chef improvements○ The recipes do too much at the moment.

What next?

● Improve Chef recipes○ The recipes do too much at the moment.○ Example: cluster management.

Questions?

Thank you!

Sam Starlingsam.starling@bbc.co.uk

Johan Hjerlingjohan.hjerling@bbc.co.uk