View
21
Download
0
Category
Preview:
Citation preview
Data Platform Airlift21 de Outubro \\ Microsoft Lisbon Experience
What’s new in the Azure Data PlatformRicardo Peres
Luis Calado
Azure DocumentDB
Azure Search
Azure Machine Learning Marketplace
Azure SQL Database
Azure Data Lake
Azure Data Factory
Agenda
Headline
Core Concepts
Resources
Indexes
Querying
Paging
Updating
Transactions
Partition Resolvers
User Defined Functions
Stored Procedures
Triggers
Security
Limits
Search
Best Practices
Headlines
NoSQL database as a service for JSON documents
Schemaless
RESTful
Part of Azure – only available online
Highly scalable
Several bindings (.NET, JavaScript, Python, ...)
Core Concepts
Resources (1 of 3)
Documents that live in DocumentDB
All have a unique addressable URL (_rid or id):https://{account}.documents.azure.com/dbs/{_rid-db}/colls/{_rid-col}/docs/{_rid-doc}
All live inside a collection
A collection lives inside a database
A database belongs to an account
A collection can take different kinds of documents
Resources (2 of 3)
Either POCOs or inherit from Resource
Some built-in properties:
If an id property is not specified, one will be provided (Guid)
Case matters!
Property User Settable Purpose
_rid No System generated, unique and hierarchical
identifier
_etag No HTTP etag required for optimistic concurrency
control
_ts No Last updated timestamp
_self No Unique addressable URL
id Yes User defined unique name
Resources (3 of 3)
Can have attachments:https://{account}.documents.azure.com/dbs/{_rid-db}/colls/{_rid-col}/docs/{_rid-doc}/attachments/{_id-attch}
Additional properties:
Property User Settable Purpose
contentType Yes The content type of the attachment
media Yes The URL link or file path where the
attachment resides
Indexes (1 of 2)
Consistency can be configured per collectionConsistent: indexes are updated synchronously
Lazy: indexes are updated asynchronously
None
Indexes (2 of 2)
By default, all paths are indexed, can be overriden
Three kinds of property indexes:Hashed: for exact matchesRange: for range comparisons, orderingSpatial: for geospatial queries
Three kinds of property value indexes (from JSON):String (precision: 1-100 or -1)Number (precision: 1-8 or -1)Point
A collection can have several indexes at once
If a collection does not have an index, it cannot be queried except by id or self link!
Querying – SQL (1 of 3)
Returns JSON
Joins only inside document (collections)
No comparison of different data types (undefined)
Math: +, -, *, /, %
Bitwise: |, &, ^, <,>>, >>>
Logical: AND, OR, NOT
Comparison: =, !=, <, >, <=, >=, <>
String: ||
Ternary and coalesce: ?, ??
IN, BETWEEN, ORDER BY
Parameterized – no SQL injection
Querying – SQL (2 of 3)
SQL functions:Math: ABS, CEILING, EXP, FLOOR, LOG, LOG10, POWER, ROUND,
SIGN, SQRT, SQUARE, TRUNC, ACOS, ASIN, ATAN, ATN2, COS, COT, DEGREES, PI, RADIANS, SIN, TAN
Type checking: IS_ARRAY, IS_BOOL, IS_NULL, IS_NUMBER, IS_OBJECT, IS_STRING, IS_DEFINED, IS_PRIMITIVE
String: CONCAT, CONTAINS, ENDSWITH, INDEX_OF, LEFT, LENGTH, LOWER, LTRIM, REPLACE, REPLICATE, REVERSE, RIGHT, RTRIM, STARTSWITH, SUBSTRING, UPPER
Array: ARRAY_CONCAT, ARRAY_CONTAINS, ARRAY_LENGTH, ARRAY_SLICE
Spatial: ST_DISTANCE, ST_WITHIN, ST_ISVALID, ST_ISVALIDDETAILED
Querying – SQL (3 of 3)
SQL Ternary and coalesce: ?, ??
SELECT (c.grade < 5)? "elementary": "other" AS gradeLevel
FROM Families.children[0] c
SELECT f.lastName ?? f.surname AS familyName
FROM Families f
Projecting into new JSON objects:SELECT { "state": f.address.state, "city": f.address.city, "name": f.id }
FROM Families f
WHERE f.id = "AndersenFamily“
Creating arrays:SELECT [f.address.city, f.address.state] AS CityState
FROM Families f
Returning single values:SELECT VALUE “Hello World”
[{ "$1": { "state": "WA", "city": "seattle" }, "$2": { "name": "AndersenFamily" } }]
[ { "CityState": [ "seattle", "WA" ] }, { "CityState": [ "NY", "NY" ] } ]
[ "Hello World" ]
Querying - LINQ
LINQ functions:Math: Abs, Acos, Asin, Atan, Ceiling, Cos, Exp, Floor, Log, Log10,
Pow, Round, Sign, Sin, Sqrt, Tan, Truncate
String: Concat, Contains, EndsWith, IndexOf, Count, ToLower, TrimStart, Replace, Reverse, TrimEnd, StartsWith, SubString, ToUpper
Array: Concat, Contains, and Count
Spatial: Distance, Within, IsValid, and IsValidDetailed
Paging
Can specify maximum number of items to retrieve
Has more results / get next results
Ordering
Updating
InsertsFrom POCOFrom StreamBatching:
Document ExplorerData Migration ToolStored Procedures
ReplacesConcurrency control from Etags
DeletesBy self link or id
Transactions
No explicit transactions
Implicit inside triggers and stored procedures – only at collection level
Partition Resolvers
Specified per database
Possibly several
Can decide on which collection a document is to be saved or retrieved from
Included:HashPartitionResolver: distribute data evenly accross collections
RangePartitionResolver<T>: when there is a “natural” ordering, such as with date and time
User Defined Functions
JavaScript-based
Exist in collections
No side effects
var regexMatchUdf = new UserDefinedFunction {
Id = "REGEX_MATCH",
Body = "function (input, pattern) {
return input.match(pattern) !== null;
};",
};
SELECT udf.REGEX_MATCH("ardo", s.Id) FROM Session s
Stored Procedures
JavaScript-based
Exist in collections
Can do batching
Implicit transactions
function (gender) {
var response = getContext().getResponse();
var collection = getContext().getCollection();
var query = 'SELECT * FROM c WHERE c.Gender= "' + gender + '"';
collection.queryDocuments(collection
.getSelfLink(), query, {},
function(err, documents, options) {
response.setBody(response.getBody() + JSON.stringify(documents));
}
);
}
Triggers
JavaScript-based
Exist in collections
Two types:Pre trigger
Post trigger
function updateTrigger() {
var request = getContext()
.getRequest();
var doc = request.getBody();
doc[‘message’] = ‘Added by trigger’;
request.setBody(doc);
}
Security
Access keys:
Master (single)
Read only (multiple)
Database users – specify use at DocumentClient level
Permissions for users over resources (resource tokens: default expiration is 1h, up to 5h):
All
Read
Resources:
Collections
Documents
Attachments
Stored procedures
Triggers
User defined functions
LimitsFeature Limit
Maximum Request Units / second / collection 2500
Maximum execution time for stored procedure
and trigger
5 s
Provisioned document storage / collection 50 GB
Maximum collections per database account* 100
Maximum document storage per database
(100 collections)*
1 TB
Maximum Length of the Id property 255 chars
Maximum request size of document and
attachment
512 KB
Maximum number of JOINs per query* 5
Number of stored procedures, triggers and
UDFs per collection*
25
Number of users per database account 500.000
Search
Based on Elasticsearch and Lucene
.NET + REST APIs
Can retrieve data from DocumentDB
Best Practices
Cache the DocumentClient instance
Choose right collection index update policy
Index only properties that will be searchable and with appropriate values – watch out for ranges
Store small documents
Measure and tune request costs
Retrieve only what you need – paging, projections
Cache self links – they never change
Use partition resolvers for distributing burden
Beware throttling!
Meet the Competition
MongoDBOpen source + support model
No joins
Aggregations
Time to live
Offline deployment
Replication
Eventual consistency
ACID transactions
Map/Reduce
Several programming languages supported
RavenDBOpen source + support model
Joins across documents
Aggregations
Expiry
Offline deployment
Replication
Eventual consistency
ACID transactions
Map/Reduce
.NET, REST
References
Query Playground: https://www.documentdb.com/sql/demo
.NET Azure DocumentDB Samples: https://github.com/Azure/azure-documentdb-net
DocumentDB Studio: https://studiodocumentdb.codeplex.com/
Azure DocumentDB Data Migration Tool: http://www.microsoft.com/en-us/download/details.aspx?id=46436
Pricing: https://azure.microsoft.com/en-us/pricing/details/documentdb/
Connecting DocumentDB with Azure Search using indexers: https://azure.microsoft.com/en-us/documentation/articles/documentdb-search-indexer/
A search-as-a-service solution allowing developers to incorporate great search experiences into applicationswithout managing infrastructure or needing to become search experts.
Type Ahead
FacetsFacets
Hit Highlighting
Spelling Mistakes
Geo-Spatial Search
Paging
Sorting & Scoring
New indexers (SQL Database and DocumentDB)
New language support (35 languages including pt-PT)
Index creation in the new Management Portal
New Regions
New APIs for index creation
•
• Distance
• Intersection
•
Full Text Search
Secure data with authentication, authorization and encryption
Extended Events
•
•
•
Azure Portal
Azure Ops Team
ML Studio
Data Scientist
HDInsight
Azure Storage
Training Set
from on-prem
Azure Portal &
ML API service
Azure Ops Team
PowerBI/DashboardsMobile AppsWeb Apps
ML API service Developer
ML Studio and the Data Scientist
• Access and prepare data
• Create, test and train models
• Collaborate
• One click to stage for
production via the API service
Azure Portal & ML API serviceand the Azure Ops Team
• Create ML Studio workspace
• Assign storage account(s)
• Monitor ML consumption
• See alerts when model is ready
• Deploy models to web service
ML API service and the Developer
• Tested models available as an url that can be called from any end point
Business users easily access results:
from anywhere, on any device
Cloud
Event Hubs
ML Studio ML API Service
Microsoft
Azure Portal
Blob Storage
ML Apps
Marketplace
ML Operationalization
ML Studio
ML Algorithms
Observation
Pattern
Theory
Hypothesis
What will happen?
How can we make it happen?
Predictive
Analytics
Prescriptive
Analytics
What happened?
Why did it happen?
Descriptive
Analytics
Diagnostic
Analytics
Top-Down
Confirmation
Theory
Hypothesis
Observation
Implement Data Warehouse
Physical Design
ETL
Development
Reporting &
Analytics
Development
Install and Tune
Reporting & Analytics Design
Dimension Modelling
ETL Design
Setup Infrastructure
Understand Corporate Strategy
Data sources
ETL
BI and analytic
Data warehouse
Gather Requirements
Business Requirements
Technical Requirements
Ingestregardless of requirements
Storein native format without
schema definition
AnalyzeUsing analytic engines
like Hadoop
Interactive queries
Batch queries
Machine Learning
Data warehouse
Real-time analytics
Devices
Store and analyse data of any kind and size
Develop faster, debug and optimise smarter
Interactively explore patterns in your data
No learning curve—use U-SQL, Spark, Hive, HBase and Storm
Managed and supported with an enterprise-grade SLA
Dynamically scales to match your business priorities
Enterprise-grade security with Azure Active Directory
Built on YARN, designed for the cloud
`
AZURE DATA LAKE
DEV
TOOLSVisual
Studio
PowerShell
MS
Azure Data Factory
Azure Stream
Analytics*
MS
HDInsight
Kona
Azure SQL
DW*
AzureML*
3rd Party
Informatica*
3rd Party
Cloudera*
Hortonworks*
MapR*
Open Source
Sqoop
Flume
MS
RevolutionR*
PowerBI*
3rd Party
TBA
PLATFORMS
APPLICATIONS
DATA INTEGRATION TOOLS
Last Name First Name Country Age …
Flasko Mike Canada 32
Anand Subbaraj USA 30
Gaurav Malhotra USA 72
… …. …. ….
Last Name First Name At risk of
churning
….
Flasko Mike Yes
Anand Subbaraj No
Gaurav Malhotra Yes
… ….
Call Log Files
Customer Table
Call Log Files
Customer Table
Customer
Churn Table
Data Sources Ingest Transform & Analyze Publish
Customer
Call Details
Customers
Likely to
Churn
Recommended