Embed Size (px)
Citation preview
Gebruikersdag 2018
Power BI Custom ConnectorsJeroen ter Heerdt
[email protected]@jeroenterheerdthttp://www.dutchdatadude.com
Power Query
Power Query in Excel
Power Query in Power BI Desktop
Power Query in SQL Server Data Tools
Power Query Editor
Power QueryDesign Tenents
Design Tenets for “M”
Power QuerySyntax and language flow
Syntax for a Simple Query
SELECT Orders.OrderDate, Products.OrderID, Products.ProductSKU
FROM Products
INNER JOIN Orders ON Products.OrderID = Orders.OrderID
ORDER BY Products.ProductSKU
from p in Products
join o in Orders on p.OrderID equals o.OrderID
orderby p.ProductSKU
select new { o.OrderDate, p.OrderID, p.ProductSKU }
Products.Join(Orders, Products,
o => o.OrderID, p => p.OrderID,
(p, o) => new { o.OrderDate, p.OrderID, p.ProductSKU }
).OrderBy( p => p.ProductSKU )
let
Joined = Table.Join( Products, "OrderID", Orders, "OrderID" ),
Columns = Table.SelectColumns(Joined, {"OrderDate", "OrderID", "ProductSKU"}),
Sorted = Table.Sort( Columns, "ProductSKU" ),
in
Sorted
Language Flow
lazy evaluation
letSource = Web.Page(Web.Contents("http://www.bing.com/blogs/site_blogs/b/search/archive/2013/12/01/eoy.aspx")),WebTable = Source{index}[Data],RenamedColumns = Table.RenameColumns(WebTable,{{"Column1", "Rank"}, {"Column2", "2013"}})
inRenamedColumns
letSource = Sql.Database("localhost", "AdventureWorksDW2012"),DimCat = Source{[Schema="dbo",Item="DimProductCategory"]}[Data],DimSubCat = Source{[Schema="dbo",Item="DimProductSubcategory"]}[Data],DimCustomer = Source{[Schema="dbo",Item="DimCustomer"]}[Data],Joined = Table.NestedJoin(DimSubCat,{"CategoryKey"},DimCat,{"CategoryKey"},"Category",JoinKind.Inner)
inJoined
How it Actually Works
letSource = Sql.Database("localhost", "AdventureWorksDW2012"),DimCat = Source{[Schema="dbo",Item="DimProductCategory"]}[Data],DimSubCat = Source{[Schema="dbo",Item="DimProductSubcategory"]}[Data],DimCustomer = Source{[Schema="dbo",Item="DimCustomer"]}[Data],Joined = Table.NestedJoin(DimSubCat,{"CategoryKey"},DimCat,{"CategoryKey"},"Category",JoinKind.Inner)
inJoined
Joined
DimSubCat
DimCat
Source
Source
Values are lazy – they are only evaluated when they are “pulled”
Power QueryQuery Folding
Query Folding (Delegation, Pushdown)
let
Source = Sql.Database("localhost", "AdventureWorksDW2012"),
Products = Source{[Schema="dbo",Item="DimProduct"]}[Data],
RemovedOtherColumns = Table.SelectColumns(Products,{"ProductKey", "Name"}),
RenamedColumns = Table.RenameColumns(RemovedOtherColumns,{{"Name", "Product"}}),
FilteredRows = Table.SelectRows(RenamedColumns, each [ProductKey] < 10),
Sorted = Table.Sort(FilteredRows,{{"ProductKey", Order.Ascending}})
in
Sorted
Mashup Engine
Folding to Query Languages like SQL
SELECT ProductKey, Name as [Product]FROM [dbo].[DimProduct]WHERE [ProductKey] < 10ORDER BY [ProductKey]
let
Source = Sql.Database("localhost", "AdventureWorksDW2012"),
Products = Source{[Schema="dbo",Item="DimProduct"]}[Data],
RemovedOtherColumns = Table.SelectColumns(Products,{"ProductKey", "Name"}),
RenamedColumns = Table.RenameColumns(RemovedOtherColumns,{{"Name", "Product"}}),
FilteredRows = Table.SelectRows(RenamedColumns, each [ProductKey] < 10),
Sorted = Table.Sort(FilteredRows,{{"ProductKey", Order.Ascending}})
in
Sorted
The Mashup Engine generates
queries in the native language of
the source, pushing down as
much work as possible to the
backend. Any remaining work is
performed locally.
Join's across sources can result in
filters or join logic being passed
from one side to the other.
Partial Folding and Compensation
Mashup Engine
Query
Query
https://sharepoint.com/.../?filter=ProductKey gt 10
File is read and processed locally
Filters pushed down as query string parameters
Rename and sort
done locally
Rename, filter,
and sort done
locally
What Gets Folded?
Power QueryCredentials and Privacy Levels
Credentials and Privacy Levels
Power QueryType system
Symbolic Operators
• Arithmetic (numbers, durations, and date-time values) +x -x x + y x - y x * y x / y
• Equality, inequality x = y x <> y
• Comparison x < y x <= y x > y x >= y
• Concatenation (text & text, list & list, record & record, table & table, date & time)
x & y
• Field access (or null) r[f] r[f]?
• Projection (fill with null) r[[f1],[f2]] r[[f1],[f2]]?
• Item lookup (or null) l{i} l{i}?
• Item lookup by key (or null) l{[k1=v1, k2=v2]} l{[k1=v1, k2=v2]}?
• Function application f(a1, a2)
• Not-implemented error …
• Self-recursive reference @n
Type System
A small set of built-in types• any, none
• null, logical, number, text, binary
• time, date, datetime, datetimezone, duration
Complex Types• list, record, table, function
Sql.Database = (server as text, database as text, optional options as nullable record) as table
MyCoolFunction = (index as number, category as text) as nullable table
Date.StartOfDay = (dateTime as any) as any
Ascribed Types
A value's ascribed type is the type to which a value is declared to conform. When a
value is ascribed a type, only a limited conformance check occurs. M does not perform
conformance checking beyond a nullable primitive type. M program authors that
choose to ascribe values with type definitions more complex than a nullable primitive-
type must ensure that such values conform to these types.
Commonly used ascribed types• Byte.Type, Int8.Type, Int16.Type, Int32.Type, Int64.Type
• Single.Type, Double.Type, Decimal.Type, Currency.Type, Percentage.Type
• Character.Type
Simple Values
SIMPLE VALUE LITERAL
Null null
Logical true, false
Number 1, 1.2, 1.2e-3, #infinity, #nan
Text "hello, world!"
Date #date(2013, 3, 8)
Time #time(15, 10, 0)
DateTime #datetime(2013, 3, 8, 15, 10, 0)
DateTimeZone #datetimezone(2013, 3, 8, 15, 10, 0, -8, 0)
Duration #duration(1, 13, 59, 12.34)
Complex Values - Lists
Complex Values - Records
Complex Values – Tables
Complex Values - Others
COMPLEX VALUE LITERAL
Function MyFunction = ( x, y, optional z ) => if z = null then
x + y else
(x + y) / z
Type type table [ n = number, #"n^2" = number ]
Binary #binary({0x68, 0x65, 0x6C, 0x6C, 0x6F})
Anatomy of a Table
letSource = Excel.Workbook(File.Contents("Sales.xlsx")),MyTable = Source{[Item="Data",Kind="Sheet"]}[Data]
inMyTable
Anatomy of a Table
MyTable[ProductKey] = { 217, 231, 485, 538, 480, 528, 480, 477 }
A column in the table is a list.
Anatomy of a Table
A row in the table is a record.
MyTable{2} = [ ProductKey = 485, OrderQuantity = 1, UnitPrice = 21.98,SalesAmount = 21.98, OrderDate = #date(2008, 1, 1) ]
Anatomy of a Table
A table is a list of records
MyTable = {[ProductKey = 217,…],[ProductKey = 231,…],…}
Unary functions
• Many library functions take functions as arguments
• Often, those parameter functions are unary• A special syntactic form helps construct unary function values
• An ‘each’ expression is just shorthand for a unary function• The single parameter of an ‘each’ function is named _
• For conciseness and to get close to the DAX syntax, the _ can be omitted when accessing fields or columns
Table.SelectRows( table, (r) => r[Manager] = r[Buddy] )
Table.SelectRows( table, each _[Manager] = _[Buddy] )
Table.SelectRows( table, each [Manager] = [Buddy] )
Special Syntactic Forms
Conditional expression
Let expression
Error expression (think: “throw”)Terminates evaluation and raises error
Try expression (think: “catch”)Attempts evaluation, optional “otherwise”
Encodes results and errors as record values
if 1 < 2 then "hurray" else "sad face"
let x = Number.Atan(3) in x * x
error "Here I go wrong"error [ Reason = "Expression.Error",
Message = "'t' should be positive", Detail = t ]
try error "Bad"// [ HasError = true,// Error = [// Reason = "Expression.Error",// Message = "Bad",// Detail = null// ]// ]try error "Bad" otherwise 42// 42
Power QueryWriting M and tips and tricks
Writing M
Advanced ViewAccess/edit the full query text in the Advanced View
The formula for the current step is shown in the formula bar
The “fx” button creates a new step, referencing the previous step by name
Case Sensitive!Functions are typically CamelCase
Library functions are usually named in the form of Type.Action
Table.SelectColumns, Number.ToText, etc.
Special syntactic forms are usually lower caseif, then, etc.
Tips and Tricks: Leverage the UI
Generate steps using the UI, then tweak the code
Some things only work through the UI (“auto steps”)
Tips and Tricks: Library Functions
Use #shared to see all exported functions (and keywords)
Typing in the function name will display its help and prompt for parameter
Tips and Tricks: Troubleshooting
Use try/catch to isolate errors
Select and remove rows with errors
Table.Buffer will stop folding from occurring
Mashup EngineExtensibility – teaching new tricks
M Extensibility
Allows definition of new functions
• Dynamically loaded at runtime
• Can be used to add new Data Source functions
•Associated with a “Data Source Kind” and credential
Extensions packaged as a zip file (.mez, or .pqx)
• Contains M definition, icons, and string resources
M Extensibility: how it works
Data Connector Extensibility
• Extend Power BI data source
options through OData/ODBC/M
• Detects service capabilities
• Sandboxed, cloud deployable
• M Extension is a lightweight
wrapper providing:
• Icons and branding
• Compensation and query
generation
• Custom authentication flows
Data
Source
Simple OData Examplesection TripPin;
// This record declares that TripPin.Feed should be recognized as data source function of kind "TripPin". The Publish record// indicates that it should be shown in the Power Query "Get Data” dialog.[DataSource.Kind="TripPin", Publish="TripPin.Publish"]shared TripPin.Feed = (url as text) =>
let// Wrapping another data source function gives us all the capabilities of that function// (Paging, OData $metadata, query folding).source = OData.Feed(url)
insource;
// Data Source Kind descriptionTripPin = [
// Declares the supported type(s) of authentication. In this case, Implicit = Anonymous web access.Authentication = [
Implicit = []],// Assigns a label to the data source credential. This will be displayed in the "Manage Data Sources" dialog.Label = "TripPin Part 1 - OData"
];
// Data Source UI publishing descriptionTripPin.Publish = [
Beta = true,Category = "Other",ButtonText = { "TripPin OData", "TripPin OData" }
];
Mashup EngineExtensibility – demo
Gebruikersdag 2018
Bedankt!Vergeet niet de evaluatie in te vullen!Scan de QR code.
