Domains - Don't leave your data model without them

An Application Developer perspective on Data Modeling

Exploring the Effective Use of Schema Names and Domains

by creating another “One to Many Relationship”

Prepared By: Peter Heller

Prepared by: Peter Heller

NYC DCAS / OMIS

Data Modeling Zone

October 22, 2014

AGENDA


• Definitions

• Domain

• Schema Name

• Embracing Domains (UDT)

• Q&A

What is a Domain?


The ANSI SQL-89 and SQL-92 standards define a CREATE DOMAIN

statement, which Microsoft Transact SQL (T-SQL) supports as user-

defined data types (UDTs) optionally combining with check constraints

and default values.

The ANSI SQL-89/92 domain is derived from existing base data types, as

in the following pseudo code examples:

.

Domain User defined Datatype

-- No Optional Schema Name

CREATE DOMAIN CustomerAccountNumber AS

VARCHAR(20)

-- Optional Schema Name Creation for further

separation and acts as an alias for an owner.

CREATE SCHEMA [udtCategory]

CREATE TYPE [udtCategory].[CustomerAccountNumber]

FROM [VARCHAR](20) NOT NULL

What is a Schema Name?


� The ANSI SQL-92 standard defines a schema as a collection of database

objects that are owned by a single principal and form a single namespace.

� All objects within a schema must be uniquely named and a schema must be

uniquely named in the database catalog.

� SQL Server 2005 breaks the link between users and schemas, users do not

own objects. Schemas own objects and principals own schemas.

� A schema can be owned by either a primary or secondary principal, with the

term “principal” meaning any SQL Server entity that can access securable

objects.

� Principle types that can own schemas:� Primary

� SQL Server Login

� Database User

� Windows Login

� Secondary

� SQL Server Roles

� Windows Groups

� Default Schemas

Better way to define Attributes based upon Domains.


• Domain and UDT (User defined Datatype) are interchangeable.

• The domain definition is an alias for a SQL Datatype on which the database

attributes are defined.

• Further, the domain definition provides a unique SQL Datatype for all application

attributes within the database and across the enterprise.

• Schema Name helps categorize domain groupings

• Traditionally, the check constraints and default values are loosely coupled.

� The check constraints and default values tightly coupled should not be

considered (i.e., sp_bindefault & sp_bindrules)

Using Domains as an integral aspect of your metadata design.Using Domains as an integral aspect of your metadata design.Using Domains as an integral aspect of your metadata design.Using Domains as an integral aspect of your metadata design.(Thinking inside or outside the box)(Thinking inside or outside the box)(Thinking inside or outside the box)(Thinking inside or outside the box)

� Domains provide categorization and reusability of SQL Datatypes.

� Domain metadata repository provides a unique grouping of your application attributes.

� Domain names can provide typed inferences for application attributes


Tool Independent Example:Schema Name, Domains and Sub-Domains

•Categorize by Schema Name • Domain (i.e., Energy Units and Demand Units)

• Sub-Domain using specific types within the Generic Domain


Using Domains with Data Modeling Tools

• Data modeling tools such as ERwin may provide a core group of base domains. In ERwin, the core group consists of five domains per model


Data Modelers are accustomed to Entity/ Attribute

Modeling using standard SQL Datatypes.

Domain Based modeling defines a relational contract

between the domain and the attribute using standard

SQL Datatypes.

Domain Based modeling defines a parent \ child

relationship between the domain and the children

attribute(s).

Domain Terms (Dictionary)

•The benefits of development of an exclusive dictionary (repository) of Domains for an entire application.

•Discrete number of application data types

•Domain based modeling manages attribute changes by modifying the domain and forcing a bubble up effect in the database.

Example:

Domain udtCommonNumber.QTY changes from decimal(8,2) to decimal(12,2) bubbles up

•The Entity is exclusively defined by the attributes leveraging

domains.


CLD20

Slide 9

CLD20 What happens if you don't wnat everything with that same domain to change?Lehn, Carol {BIS}, 4/25/2014

Evolution of Data ModelingFrom EAR to ERD

• When I started working with databases, it was called EAR modeling for Entity, Attributes & Relationships Modeling.

• Over time, the attribute was removed from the acronym in favor of Entity Relationship Modeling.


My Evolving Migration to Using DomainsMy Evolving Migration to Using DomainsMy Evolving Migration to Using DomainsMy Evolving Migration to Using Domains

1. Started out not using domains

2. Began using domains with tight coupling (SQL Server 2005)

• Microsoft 2012 Plus Deprecated features tight coupling features

• sp_bindefault

• sp_bindrules

3. Moved to using domains loosely coupling defaults and rules.


Tight Tight Tight Tight Coupling through Coupling through Coupling through Coupling through Encapsulation of the UDT, Rules & DefaultsEncapsulation of the UDT, Rules & DefaultsEncapsulation of the UDT, Rules & DefaultsEncapsulation of the UDT, Rules & Defaults

�The application column would inherit the

�Datatype

• default value though sp_bindefault

• rule through sp_bindrules

�The default and rule from the UDT can still be tightly coupled through check constraint or add default

�You may also loosely couple by decoupling the default and/or rule from the domain definition.

�Simulate the create default and create rule through User Defined scalar Functions (UDF) in the

next slide.


Loosely Coupling without Encapsulation of the UDT, Rules & Defaults


Loose

Coupling

Loose

Coupling

Example using Erwin Data Modeler to create rules and defaults


• Microsoft has deprecated sp_bindrules and sp_bindefault.

� Check constraint replaces sp_bindrules.

� Check Default replaces sp_bindefault.

� Both check constraint and check default are added as inline code to each

attribute

Slide 14

CLD22 What is this being replaced with?Lehn, Carol {BIS}, 4/25/2014

Domain based Table

DadFirstname udtStringCommon.firstName

DadLastname udtStringCommon.lastName

MomFirstname udtStringCommon.firstName

MomLastname udtStringCommon.lastName

KidFirstname udtStringCommon.firstName

KidLastname udtStringCommon.lastName

KidFirstname2 udtStringCommon.firstName

KidLastname2 udtStringCommon.lastName

Homephone udtStringCommon.telephone

Homephone2 udtStringCommon.telephone

Workphone udtStringCommon.telephone

Workphone2 udtStringCommon.telephone

Workphone3 udtStringCommon.telephone,

fax udtStringCommon.telephone

cellphone udtStringCommon.telephone

blackberry udtStringCommon.telephone,

Non-Domain based Table

DadFirstname varchar(10),

DadLastname varchar(10),

MomFirstname varchar(10),

MomLastname varchar(10),

KidFirstname varchar(10),

KidLastname varchar(10),

KidFirstname2 varchar(10),

KidLastname2 varchar(10),

Homephone varchar(10),

Homephone2 varchar(10),

Workphone varchar(10),

Workphone2 varchar(10),

Workphone3 varchar(10),

fax varchar(10),

cellphone varchar(10),

blackberry varchar(10)


Exaggerated Example

Using varchar (10) as a singular Datatype versus

SchemaName.DomainName definition

• SQL Domain examples help differentiate the meaning and usage of varchar(10)

• Create domain udtStringCommon.Telephone varchar(10)

• Create domain udtStringCommon.LastName varchar(10)

• Create domain udtStringCommon.FirstName varchar(10)


Domain (User Defined Datatype)


Domain Based Generic Lookup Table Example

Behind the scenes SQL pseudo code :

CREATE TABLE [Subsystem].[GenericLookupTable]

[GenericLookupTableSurrogateKey] [udtCommon].[SurrogateKey] NOT NULL,

[Acronym] [udtCommon].[Acronym] NOT NULL,

[ShortDescription] [udtCommon].[ShortDescription] NOT NULL,

[LongDescription] [udtCommon].[LongDescription] NOT NULL,

[Notes] [udtCommon].[Notes] NULL,

[AuthenticatedUser] [udtCommon].[AuthenticatedUser] NOT NULL,

[DateAdded] [udtCommon].[DataAdded] NOT NULL,

[DateLastUpdated] [udtCommon].[DateLastUpdated] NOT NULL,

Important note: SQL includes the Schema

name as part of the fully

qualified Domain name.


Erwin Domain Based Generic Lookup Table Example

1. In the table layout, the UDT is shown without the owner

2. The domain editor shows the physical name and the

schema owner.

3. The domain parent shows a hierarchical view of

domains and their sub-domains. It does exclude the

schema owner.

1. In the table layout, the UDT is shown without the owner

2. The domain editor shows the physical name and the

schema owner.

3. The domain parent shows a hierarchical view of

domains and their sub-domains. It does exclude the

schema owner.

Domain usage Example of Type Inference

At previous NYMUG presentations, the entity level was the foundation of all of the base models. Questions would arise such as the following:

“How does this attribute correspond to the attribute on another system?” was common complaint (i.e., Is the CustId the same as CustomerNumber?

A pseudo code Solution:• Create domain udtCommon.Udt.AccountNumber varchar(15)

�Declare CustId udtCommon.AccountNumber�Declare CustomerNumber udtCommon.AccountNumber

Utilizing the domain, an inference could easily be made that both attributes were based upon udtCommon.Udt.AccountNumber and they were both account numbers.


Creating Defaults with scalar functions?


Check with your DBA:

1. Scalar functions are self documenting. It may be a performance hit over inline

code.

2. Provides encapsulation of code and reuse.

No Parameter Function Parameterized Function

Create FUNCTION [Defaults].[udf_GetDateNow] ( )

RETURNS DATETIME

AS

BEGIN

RETURN GETDATE();

END;

Create FUNCTION [Defaults].[udf_StartingRangeValue]

(

@StartingRangeValue INT

)

RETURNS INT

AS

BEGIN

RETURN @StartingRangeValue;

END;

Inline code Scalar Function

Erwin Examples: Creating Defaults



1. Scalar functions are self documenting. It may be a performance hit over inline code.


Loosely Coupling the Default by Adding a Constraint

Default using Default Scalar Function?



Syntax: Alter Table ADD CONSTRAINT

[DF_GenericLookupTable_RangeValueFunction] DEFAULT

ALTER TABLE [Subsystem].[GenericLookupTable]

ADD CONSTRAINT [DF_GenericLookupTable_RangeValueFunction]

DEFAULT ([Defaults].[udf_StartingRangeValue]((1))) FOR

[RangeValueFunction]


ADD CONSTRAINT [DF_GenericLookupTable_DateAdded]

DEFAULT ([Defaults].[udf_GetDateNow]())

FOR [DateAdded]

Loosely Coupling the Default by Adding a

Constraint Default using Default Inline code?



Syntax: Alter Table ADD CONSTRAINT

[DF_GenericLookupTable_RangeValue1] DEFAULT


ADD CONSTRAINT [DF_GenericLookupTable_RangeValue1]

DEFAULT ((1))

FOR [RangeValueInlineCode]


ADD CONSTRAINT [DF_GenericLookupTable_DateAdded]

DEFAULT ((GetDate()))

FOR [DateAdded]

Loosely Coupling the Default Adding a Constraint Default?



Alter Table ADD CONSTRAINT [DF_UniqueName_Counter] DEFAULT

Default Inline Code

Loosely Coupling the Default Adding a Constraint Default?



Alter Table ADD CONSTRAINT [DF_UniqueName_Counter] DEFAULT

Default using Scalar function

Creating Rules with scalar functions?




code.


Example One Example Two

Create FUNCTION [Rules].[udf_ValidateRangeStartToEnd]

(

@Col INT ,

@StartingRangeValue INT ,

@EndingRangeValue INT

)

RETURNS TINYINT

AS

BEGIN

DECLARE @Status INT = 0;

IF ( @StartingRangeValue <= @Col

AND @Col <= @EndingRangeValue

)

SET @Status = 1

ELSE

SET @Status = 0

RETURN @Status;

END;

Create FUNCTION [Rules].[udf_ValidateList]

(

@Col VARCHAR(128) ,

@ValidList VARCHAR(MAX)

)

RETURNS TINYINT

AS

BEGIN


SELECT @Status = CHARINDEX(@Col, @ValidList)

IF ( @Status <> 0 )

SET @Status = 1

RETURN @Status;

END;

Erwin Example: Scalar function




code.


Example One Example Two

Loosely Coupling the Rule Adding a Constraint Check using a scalar function?


Check with your DBA:Alter Table with nocheck

ADD CONSTRAINT[CK_RangeStartToEnd_1] CHECK


WITH NOCHECK ADD CONSTRAINT [Rule_RangeStartToEnd_1]

CHECK

(([Rules].[udf_ValidateRangeStartToEnd]([RangeValueFunction],(1),(10))=(1)))


WITH NOCHECK ADD CONSTRAINT

[Rule_ValidateList_1]

CHECK (([Rules].[udf_ValidateList]([ListFunction],'ABC,DEF,GHI,KLM')=(1)))

Loosely Coupling the Rule Adding a Constraint Check using a Inline code?


Check with your DBA:Alter Table with nocheck

ADD CONSTRAINT[CK_RangeStartToEnd_1] CHECK


WITH NOCHECK ADD CONSTRAINT [CK_RangeStartToEnd_1]

CHECK (((1)<=[RangeValueInlineCode] AND [RangeValueInlineCode]<(11)))


WITH NOCHECK ADD CONSTRAINT

[CK_ValidateList_1]

CHECK ((charindex([ListInlineCode],'ABC,DEF,GHI,KLM')>(0)))

Erwin Example: Loosely Coupling the Rule by Adding a Constraint Check ?


Check with your DBA:Alter Table with nocheck ADD CONSTRAINT [Some Constraint] CHECK

Inline Code

Function Code

Function Code

Erwin Example: Loosely Coupling the Rule by Adding a Constraint Check ?


Check with your DBA:Alter Table with nocheck ADD CONSTRAINT [Some Constraint] CHECK

Inline Code

Inline Code

Function Code

Creating Cross Object Rules with scalar functions?



1. Scalar functions are self documenting. It may be a performance hit over inline code.


Create FUNCTION [Rules].[udf_XcrossValidateFromDateLessThanorEqualToDate]

(

@FromDate DATETIME ,

@ToDate DATETIME

)

RETURNS TINYINT

AS

BEGIN


SET @Status = 0;

IF ( @FromDate IS NOT NULL

AND @ToDate IS NULL

)

RETURN 1;

IF ( @FromDate <= @ToDate )

RETURN 1;

RETURN @Status;

END;

Metadata reporting leveraging the view


Select * from [Utils].[uvw_FindColumnDefinitionPlusDefaultAndCheckConstraint]

provides the list below.

Additional reporting will be included with the presentation.

• PowerQuery and PowerPivot in Excel 2013

• SQL Server Reporting Services Report

• Straight export of data into Excel 2013

Leveraging the view across the Enterprise to develop a

Metadata Domain Repository for Analysis purposesMetadata Domain Repository for Analysis purposesMetadata Domain Repository for Analysis purposesMetadata Domain Repository for Analysis purposes


Adding three more columns to the view ServerName, DatabaseName, DatabaseContact

[Utils].[uvw_FindColumnDefinitionPlusDefaultAndCheckConstraint] below.

INSERT INTO Metadata.Domain Repository

(ServerName, DatabaseName, DatabaseContact, FullyQualifiedTableName, SchemaName, TableName,

ColumnName, OrdinalPosition, FullyQualifiedDomainName, DomainName, DataType, IsNullable,

DefaultName, DefaultNameDefinition, CheckConstraintRuleName, CheckConstraintRuleNameDefinition

)

SELECT @@SERVERNAME AS ServerName, 'UdtAdventureWorks2012' AS DatabaseName,

'Steve Hoberman' AS DatabaseContact,FullyQualifiedTableName, SchemaName, TableName,

ColumnName,OrdinalPosition, FullyQualifiedDomainName, DomainName, DataType, IsNullable,

DefaultName, DefaultNameDefinition, CheckConstraintRuleName, CheckConstraintRuleNameDefinition

FROM Utils.uvw_FindColumnDefinitionPlusDefaultAndCheckConstraint

Metadata extensions that leverage the view


Just one example of extending the domain by loosely associating metadata tables with specific

functionality:

• Client side (JavaScript) Web applications seek to leverage HTML5, CSS3, Bootstrap 3,

AngularJS, Knockout, etc. for two binding.

• All application column names could be used for display labels and headings. (

Converted through regex expressions to place spaces between the Camel case letters)

Leveraging the view across the Enterprise one database at a time


1. Export the query to excelI. Create Table Schema names to fully qualify the table groupings by name in excel

II. Create Domain Schema names to segregate the domains by hierarchy in excel

a) Commonb) As granular as you deem necessary

2. Import data back into temp or scratch table

3. Select distinct concat('create schema ',[DomainSchemaName])

FROM [Scratch].[UDT_Definitions]

4. Select distinct concat('CREATE TYPE ',FullyQualifiedDomainName,' from

',DataType)

FROM [PSR].[Scratch].[UDT_Definitions]

5. Select distinct concat('alter table ',FullyQualifiedTableName,' alter column',

ColumnName,' ',FullyQualifiedDomainName,

case when IsNullable='YES' then ' null' else ' not null' end)

FROM [Scratch].[UDT_Definitions]

Example of Leveraging the view before and after Applying UDT’s


Before:

CREATE TABLE [dbo].[UploaderFiles]([FileId] [int] IDENTITY(1,1) NOT NULL,

[UserId] [int] NOT NULL,

[Description] [nvarchar](200) NOT NULL,

[UploadTime] [datetime] NOT NULL,

[FileGuid] [uniqueidentifier] NOT NULL,

[FileSize] [int] NOT NULL,

[FileName] [nvarchar](200) NOT NULL,

[TempFileName] [nvarchar](200) NOT NULL,

[IPAddress] [nvarchar](20) NOT null)

After:

CREATE TABLE [Uploader].[Files]([FileId] [Common].[SurrogateKey] IDENTITY(1,1) NOT NULL,

[UserId] [Common].[SurrogateKey] NOT NULL,

[Description] [Common].[FileName] NOT NULL,

[UploadTime] [Common].[DateYYYYMMDD] NOT NULL,

[FileGuid] [Common].[SurrogateKeyUUID] NOT NULL,

[FileSize] [Common].[ByteCount] NOT NULL,

[FileName] [Common].[FileName] NOT NULL,

[TempFileName] [Common].[FileName] NOT NULL,

[IPAddress] [Common].[IPAddress] NOT null)

Example of Leveraging the view distribution of UDT’s


Created using PowerPivot

Conclusion


1. Provide an awareness of domain based modeling.

2. Be as granular as you deem necessary.

3. Be extremely verbose in naming your objects; � Inform the developers about intellisense if there is an issue

4. Work with your DBA as to which techniques are

preferred for your physical deployment.

Final Suggested Design Considerations WhenLeveraging Domains(Tool Independent)

� Develop Standards that are easily maintainable and extensible� Naming Conventions for all objects

� Pascal case� Verbose Names

� Domain based application attributes � Explicit Domain Names for universal inference

� Change Datatypes “en masse” selectively� Identify all application attributes available for change� Add or drop Check Constraints� Add or drop Default values

� Categorize Data into Hierarchical groupings� Sub-categorize domains through SchemaNames � Further sub-categorization of Datatypes (i.e. Choice lists, etc.)

� Fully Qualified Entity Names� Aggregation of Entity groupings through Schema Names� Use prefixes on the Entity instead of at the attribute level. � The column name should not constrict the possibility of future datatype

changes


A Special Thanks To


�Data Modeling Zone

�Steve Hoberman

�New York Model Users Group

�Ben Ettlinger

�Carol Lehn

Microsoft’s Data Quality Services Toolleverages domain management

FYI only


\

CLD2

Slide 42

CLD2 What is this screenshot from? I think labeling it in small print would be a good idea for people who don't see the presentation and as

a reminder for those who do see itLehn, Carol {BIS}, 4/25/2014

Technology

Domains - Don't leave your data model without them