Embed Size (px)
DESCRIPTION
Presented at DevIntersection / AngleBrackets 2014. I showed how to set up, develop and run NoSQL solutions for the cloud on Windows and Linux using Windows Azure. Also show you how to build multi-tier applications in the cloud that access NoSQL data. This session included an introduction to our Platform-as-a-Service offerings for MongoDB and CouchDB, as well as prepackaged Linux VMs that run Cassandra, Riak, Redis and other NoSQL data stores with a few clicks. We’ll also introduce you to the Developer Centers for Windows Azure, the Azure SDKs, our selection of plugins for popular open source developer tools, DevOps services, and other tools and materials we’ve developed to make life easier for application developers.
Citation preview
Developing Open Source
Solutions with NoSQL and
Other Cloud Services on Azure
Brian Benz, Sr. Technical Evangelist
Microsoft Open Technologies, Inc.
2 http://www.anglebrackets.org
Agenda
Azure
NoSQL
NoSQL on Azure
MongoDB and CouchDB Options
NoSQL on Azure IaaS, PaaS, SaaS
Azure Web Sites
Q&A
Azure
Flexible
Azure IaaS, PaaS, SaaS
Azure Virtual Machines – OS Options
Windows Server 2008 and 2012
OpenSUSE and SUSE Enterprise
CentOS
Ubuntu
Oracle Linux
vmdepot.msopentech.com
7 http://www.anglebrackets.org
NoSQL
Alternative to traditional relational DBMS
Flexible schema
Quicker/cheaper to set up
Massive scalability
Relaxed consistency = higher performance & availability
No declarative query language = more programming
Relaxed consistency = fewer guarantees
8 http://www.anglebrackets.org
NoSQL Applications
Document: JSON, XML, other semistructured formats
Operations: Insert(), Fetch(key), Update(key, doc), Delete(key)
(Key, Value) pairs
Operations: Insert(key,value), Fetch(key), Update(key), Delete(key)
Implementation: efficiency, scalability, fault-tolerance
Records distributed to nodes based on key
Replication
Single-record transactions, “eventual consistency”
9 http://www.anglebrackets.org
New Terms
RDBMS NoSQL
Database Database
Table Collection
View, Row Document
Index Index
Join Embedding, Linking
Partition Shard
Partition Key Shard Key
10 http://www.anglebrackets.org
Example – Posts, Categories, Tags, and Comments
11 http://www.anglebrackets.org
Basic Document Structure{
"_id" : ObjectId("508d27069cc1ae293b36928d"),
"title" : "This is the title",
"body" : "This is the body text.",
"tags" : [
“tag 1"
],
"created_date" : ISODate("2012-10-28T12:41:39.110Z"),
"author_id" : ObjectId("508d280e9cc1ae293b36928e"),
"category_id" : ObjectId("508d29709cc1ae293b369295"),
"comments" : [
{
"subject" : "This is comment 1",
"body" : "This is the body of comment 1.",
"author_id" : ObjectId("508d345f9cc1ae293b369296"),
"created_date" : ISODate("2012-10-28T13:34:23.929Z")
},
]
}
12 http://www.anglebrackets.org
Less Basic Document Structure
{
"_id" : ObjectId("508d27069cc1ae293b36928d"),
"title" : "This is the title",
"body" : "This is the body text.",
"tags" : [
ObjectId("508d35349cc1ae293b36929c")
],
"created_date" : ISODate("2012-10-28T12:41:39.110Z"),
"author_id" : ObjectId("508d280e9cc1ae293b36928e"),
"category_id" : ObjectId("508d29709cc1ae293b369295"),
"comments" : [
ObjectId("508d359a9cc1ae293b3692a2")
]
}
13 http://www.anglebrackets.org
The Landscape - NoSQL on Azure
MongoDB
Also MongoLab MongoDB-as-a-Service
Also MongoDB Inc. MongoDB-as-a-Service (beta)
CouchDB
Cloudant CouchDB-as-a-Service
Cassandra
Riak
Redis
14 http://www.anglebrackets.org
What is MongoDB?
Open Source, by MongoDB, Inc.
Community and Enterprise Editions
Language: c++
Data model: Using BSON (binary JSON)
Replication and Sharding
Azure VMS, PaaS, and SaaS
15 http://www.anglebrackets.org
http://docs.mongodb.org/ecosystem/drivers/
• C, C++
• C#
• Go
• Erlang
• Java
• JavaScript
• Node.js
• Perl
• PHP
• Python
• Ruby
• Scala
16 http://www.anglebrackets.org
Support and Alerts
Custom Admin Tools
Backup and Recover
Performance Analysis – Slow Query Analyzer
Monitoring and Support
17 http://www.anglebrackets.org
MongoDB in Beta
19 http://www.anglebrackets.org
What is CouchDB?
Apache Project
Language: Erlang
Data model: Using JSON and REST
Map/Reduce Views and Indexes
Replication
Azure VMs, PaaS, SaaS
20 http://www.anglebrackets.org
Managed, hosted fork of CouchDB
Incremental MapReduce engine
Horizontal scaling & built-in Lucene search
8 committers to Apache CouchDB
13,000+ users
Apache CouchDB features
BigCouch: Horizontal scaling framework based on Amazon Dynamo paper
Fauxton: Futon replacement modular CouchDB Web dashboard
live JSON editor with JavaScript hinting
20
21 http://www.anglebrackets.org
What is Cassandra?
Apache Project
Language: Java
Data model: Thrift & custom binary
Fast Writes, Memory management
Replication
Azure VMs, PaaS
22 http://www.anglebrackets.org
What is Redis?
Open Source
Language: c
Data model: RESP (REdis Serialization Protocol)
Key/Value
In Memory
Persistence via snapshotting or journaling to disk
Azure VMs, PaaS
23 http://www.anglebrackets.org
What is Riak?
Open Source
Language: Erlang & C, some JavaScript
Data model: HTTP/REST or custom binary
Key/Value
In Memory
Persistence via snapshotting or journaling to disk
Azure VMs, PaaS, SaaS
Replica set hosted
on Azure
worker roles or
virtual machines
Replica Set
DriverYour
Application
Replica Sets: High Availability
Shards, config servers and mongos (router) hosted
on Azure worker roles or virtual machines
Key range A-D
Sharding: Scalability
Key range E-G Key range G-Z
Mongos
(router)Your
Application
Config 1
Config 2
Config 3
Mongos
(router)
26 http://www.anglebrackets.org
Deploying to Virtual Machines
Compared to worker role (PaaS), VM-basedreplica sets offer more control, aswell as more responsibility
MongoDB Installer for Azure is a great learning tool. Not typically used for production deployment. Puts MongoDB on OS disk instead of attached disk
Limited to 30gb, not performant for writes, Windows-only http://www.mongodb.org/display/DOCS/MongoDB+on+Azure+VM+-
+Windows+Installer
.\deploy-mongo.ps1 <node-count> <dns-prefix> <image-name> <password> <location> <pub-settings-file-path> [replica-set-name]
For Linux VM deployment, see tutorial -http://www.mongodb.org/display/DOCS/MongoDB+on+Azure+VM+-+Linux+Tutorial
azure topic verb options
Command Line Syntax Overview
prompt>
accountaccount locationaccount affinity-groupvmvm diskvm endpointvm imageserviceservice certsiteconfig
downloadimportlistshowdeletestartrestartshutdowncapturecreateattachdetachbrowseset
usernamepassworddns-prefixvm-namelb-porttarget-image-namesource-pathdisk-image-namesize-in-gbthumbprintvalue-v-vv
28 http://www.anglebrackets.org
Example: Provisioning a Replica Set
azure vm create mongoc9 "OpenLogic__OpenLogic-CentOS-62-20120531-en-us-30GB.vhd" username password -l "West US" -e
azure vm create mongoc9 "OpenLogic__OpenLogic-CentOS-62-20120531-en-us-30GB.vhd" username password -l "West US" -e 23 -c
azure vm create mongoc9 "OpenLogic__OpenLogic-CentOS-62-20120531-en-us-30GB.vhd" username password -l "West US" -e 24 -c
azure vm endpoint create mongoc9 27017 27017
azure vm endpoint create mongoc9-2 27018 27018
azure vm endpoint create mongoc9-3 27019 27019
To set up MongoDB, SSH into each instance and:
get mongodb binaries and install
create db dir
start mongod, initialize replica set
Automation Support
Capabilities
• Fully Customize VM with Data Disks and Endpoint Configuration
• Automate Virtual Network Settings
31 http://www.anglebrackets.org
32 http://www.anglebrackets.org
Conference Scheduler
Attendees can track sessions, vote sessions up and down
Offline Capabilities
IndexedDB
CouchDB VM for Data Storage
Web site hosting the application
Node.js
Grunt
Demo Architecture
35 http://www.anglebrackets.org
Docs - azure.microsoft.com/documentation/
36 http://www.anglebrackets.org
SDKs - azure.microsoft.com/downloads
37 http://www.anglebrackets.org
CLIs - azure.microsoft.com/downloads
38 http://www.anglebrackets.org
OSS DevOps - msopentech.com/projects
39 http://www.anglebrackets.org
Where to learn more
Tutorial: Building a multi-tier cloud application on Windows Azure using MongoLab, node.js, and Socket.IO
http://msopentech.com/blog/2013/09/18/tutorial-building-multi-tier-cloud-application-windows-azure-using-mongolab-node-js-socket-io/
Tutorial: Using MongoDB, Windows Communication Foundation, OData, and PowerBI to build a Business Intelligence solution on Windows Azure
http://msopentech.com/blog/2014/02/19/tutorial-using-mongodb-windows-communication-foundation-odata-powerbi-build-business-intelligence-solution-windows-azure/
Tutorial – Building a Multi-Tier Windows Azure Web application use Cloudant’s Couchdb-as-a-Service, node.js, CORS, and Grunt
http://msopentech.com/blog/2013/12/19/tutorial-building-multi-tier-windows-azure-web-application-use-cloudants-couchdb-service-node-js-cors-grunt-2/
40 http://www.anglebrackets.org
Where to learn more
MongoDB on Azure VM -http://www.mongodb.org/display/DOCS/MongoDB+on+Azure+VM
Install MongoDB on a virtual machine running CentOS Linux in Azure:http://www.windowsazure.com/en-us/manage/linux/common-tasks/mongodb-on-a-linux-vm/
Install MongoDB on a virtual machine running Windows Server in Azure:http://www.windowsazure.com/en-us/manage/windows/common-tasks/install-mongodb/
Node.js Web Application with Storage on MongoDB (Virtual Machine):http://waweb.windowsazure.com/en-us/develop/nodejs/tutorials/website-with-mongodb-(mac)/
Create a Azure web site that connects to MongoDB running on a virtual machine in Azure:http://www.windowsazure.com/en-us/develop/net/tutorials/website-with-mongodb-vm/
41 http://www.anglebrackets.org
Other Resources
Microsoft Open Technologies Blog: http://blogs.msdn.com/b/interoperability VM Depot:
http://vmdepot.msopentech.com Windows Azure development:
www.WindowsAzure.com SDKs and Command Line Tools: http://www.windowsazure.com/en-us/downloads/ Windows Azure Training Kit:
http://windowsazure-trainingkit.github.com/ Microsoft Web Platform Installer: http://www.microsoft.com/web/downloads/platform.aspx Web Matrix http://www.microsoft.com/web/webmatrix/ Project Kudu https://github.com/projectkudu/kudu The application I showed today https://github.com/axemclion/conference
Q&A
