DAT307

7/27/2019 DAT307

1/34

7/27/2019 DAT307

2/34

DENALI SSIS AND DATAQUALITY ENHANCEMENTS

Dr Greg Low

Principal Mentor and CEO

SolidQ Australia

SESSION CODE: DAT307

(c) 2011 Microsoft. All rights reserved.

7/27/2019 DAT307

3/34

Who is Greg?

CEO & Principal Mentor SolidQ Australia

Host of SQL Down Under Podcast

Microsoft RD and SQL Server MVP, MCM

PASS Regional Mentor

Author

7/27/2019 DAT307

4/34

Agenda

Designer Enhancements

Project System and ParametersSSIS Server

Data Quality Services


7/27/2019 DAT307

5/34

DESIGNER ENHANCEMENTS


7/27/2019 DAT307

6/34

Visual Studio Integration

New WPF based designer

CTP3 -> Visual Studio 2010 shell

Also in Visual Studio 2010 shell Business Intelligence Semantic Model Designer (BISM)

Analysis Services -> BISM import template

PowerPivot -> BISM import template

7/27/2019 DAT307

7/34

General Usability

UI redesign and improved visuals

Undo and redo

Selectable levels of undo/redo

Getting started pane

Zoom control

Pan control (not till CTP3)Precedence constraint label improvements

7/27/2019 DAT307

8/34

Toolbox Enhancements

Component groupings Task

Transform

New icons

Auto-detection of new toolbox items

7/27/2019 DAT307

9/34

Reliability/Development Enhancements

Many controls now more reliable and substantialreduction in memory usage Merge

Merge Join

Particularly applies to situations with Multiple inputs

Different data arrival rates on each input

New properties/methods added to allow thissame functionality in custom components

7/27/2019 DAT307

10/34

Data Flow Enhancements

Component grouping in Data Flow

Assistants Source Assistant

Destination Assistant

Data Correction Transformation

Easier options for adding/removing data viewers

Edit a components properties with inputdisconnected Easier options for correcting input mapping issues

7/27/2019 DAT307

11/34

DEMONSTRATION

Designer Enhancements


7/27/2019 DAT307

12/34

PROJECT SYSTEM ANDPARAMETERS


7/27/2019 DAT307

13/34

Package and File Format Changes

Old package format was not well-formed XML Still supported for import

Differencing was difficult

New package file is much simplified Appropriate XML structure

Attributes are now XML attributes, not separate nodes

Much more human-readable

Items within groups alphabetical References are now full paths rather than reference IDs

Readable but discardable layout section

7/27/2019 DAT307

14/34

Project System Changes

Top level object is now a Project Contains one or more packages

Is deployable along with packages

Default build now creates an ISPAC file Contains packages, project, and configurations

Two project modes Project mode (new ISPAC behaviour)

Legacy mode (2005/8 behaviour) Change via right-click option on project in Solution Explorer

Opening a 2008 project starts upgrade wizard

7/27/2019 DAT307

15/34

Project Data Sources and DSVs

Were not deployed in 2005/8

Design time objects only

No longer part of a project Except in legacy mode

7/27/2019 DAT307

16/34

Parameters

Projects now support parameters Available to all packages within the project Name and data types

Visible in expression builder and property windows Like read-only variables in a specific namespace

Always have default values at design time Optional use design time default if not specified Required must be supplied when deployed as no default exists

Environments Named groups of values assigned to groups of packages Production, Staging, Test, Development, etc.

7/27/2019 DAT307

17/34

Deployment

Right-click deployment supported from VS

ISDeploymentWizard.exe for non-VS deployments

Parameters can (or must) be configured atdeployment

7/27/2019 DAT307

18/34

DEMONSTRATION

Project System and Parameters


7/27/2019 DAT307

19/34

SSIS SERVER


7/27/2019 DAT307

20/34

SSIS Server

No right/wrong decision today on where to deploy to Filesystem SQL Server (msdb)

Integration Services node in SSMS SSIS now has its own DB (SSISDB) Packages are now deployed to the SSISDB Management enhancements

Remote execution options Execution dashboard Improved logging options (can be set at the project level)

7/27/2019 DAT307

21/34

SSIS Server Programmability

SSIS node in SSMS Catalog not auto-created at setup time

Right-click Create Catalog option on SSIS node

SQL CLR integration must be enabled to create catalog Public stored procedures (Called SFC-based API)

Deployment

Configuration

Management

Execution

Example -> packages can be run via SSMS options

Calling SSISDB.Catalog.Start_Package system stored procedure

7/27/2019 DAT307

22/34

Environments

Hierarchy

Catalog -> Folder -> Project -> Environments

Environments

Are collections of server variables

Can be mapped to projects, then server variables are

mapped to project or package parameters Can be selected at deployment or run time

7/27/2019 DAT307

23/34

DEMONSTRATION

SSIS Server


7/27/2019 DAT307

24/34

DATA QUALITY SERVICES


7/27/2019 DAT307

25/34

Its all about quality

7/27/2019 DAT307

26/34

Is Data Suitable For Business Use?

Data Quality Issue Sample Data Problem

Standard Are data elementsconsistently defined and

understood ?

Gender code = M, F, U in one system

and Gender code = 0, 1, 2 in another

system

Complete Is all necessary data present ? 20% of customers last name is blank,50% of postcodes are 9999

Accurate Does the data accuratelyrepresent reality or a

verifiable source?

A Supplier is listed as Active but went

out of business six years ago

Valid Do data values fall withinacceptable ranges?Salary values should be between60,000-120,000

Unique Data appears several times Both John Ryan and Jack Ryan appearin the system are they the same

person?

7/27/2019 DAT307

27/34

7/27/2019 DAT307

28/34


Based on a Data Quality Knowledge Base (DQKB)Knowledge-Driven

Data Domains capture the semantics of your data

Knowledge Discovery Acquires additional knowledge the more you use it

Semantics

Support use ofuser-generated knowledge and IP by3rd party reference data providers

Open and Extendible

Compelling user experience designed for increasedproductivity

Easy to use

7/27/2019 DAT307

29/34

SSIS Integration with DQS

Knowledge

Base

Reference Data Definition

Values/Rules

SSIS Data Flow

Source +

Mapping

Data correction

Component

SSIS PackageDestination

New Records

Corrections

Suggestions

Correct

Invalid

7/27/2019 DAT307

30/34

DEMONSTRATION



7/27/2019 DAT307

31/34

Session Summary

Strong Enhancements in SSIS for Denali

DQS enhances business value

Start trying it today with CTP3!

7/27/2019 DAT307

32/34

Thanks!

[email protected]

www.sqldownunder.com

http://sqlblog.com/blogs/greg_low

www.solidq.com.au

7/27/2019 DAT307

33/34

Enrol in Microsoft Virtual Academy Today

Why Enroll, other than it being free?The MVA helps improve your IT skill set and advance your career with a free, easy to accesstraining portal that allows you to learn at your own pace, focusing on Microsofttechnologies.

What Do I get for enrolment?

Free training to make you become the Cloud-Hero in my Organization Help mastering your Training Path and get the recognition

Connect with other IT Pros and discuss The Cloud

Where do I Enrol?

www.microsoftvirtualacademy.com

Then tell us what you think. [email protected]
http://www.microsoftvirtualacademy.com/Home.aspx?WT.mc_id=otc-n-au-jtc-DPR-40787http://www.microsoftvirtualacademy.com/Home.aspx?WT.mc_id=otc-n-au-jtc-DPR-40787

7/27/2019 DAT307

34/34

Documents

DAT307