Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
BI ALM & TFS
Microsoft Business Intelligence Application Life-Cycle Management
with Team Foundation Server.
(Good job there are acronyms)
About me: Dmytro Andriychenko
• Business Intelligence Architect, Developer, Data Scientist, etc.
• SQL Server BI Certified Expert
• Over 10 years data of data integration and analysis experience
• Lately focused on BI architecture, design and delivery
• Dmytro, Dmitriy, Dimitriy or just D – perfectly fine, I do not bite.
• Data United Ltd – all things data• Integrity
• Performance
• Insight
Why ALM: What can possibly go wrong?
• Knight’s tale: losing $172,222 per minute after a bad deployment
• Gartner: 40% of critical system outages are due to change management or deployment
• Career-limiting events:
• … year-end finance report figures were wrong?
What is successful ALM?
• Continuous, transparent, reliable integration and deployment
• Dev-Ops cooperation and collaboration
• Productive Agile Development
• Effective, sustainable management of• Work load/items
• Teams
• Product pipeline
• Infrastructure
Application Life-Cycle Management (ALM)
• Operate• Plan
• Deploy• Deliver
BI &DB Projects,
MS Test,Version Control
MsBuildTest Manager,
Release Manager
Bug Tracking
Backlog planning,
Sprint Board,BurnDown
chart
Sprint MonitorDev Ops
ALM I: What Planning? We are all Agile, right?
ALM I: Planning: the most important step
• Prior Proper Planning Prevents Poor Performance (Brian Tracy:10/90)
• Organise work into Features, PBIs(User Stories) and Tasks:
• Feature – project level item • Product Backlog Item (PBI): scrum – level container of work tasks
• Task: individual work item, user story, 2 days effort max
• Assign tasks to developers at the beginning of sprint & set duration
• Monitor and update tasks via task board & burn-down chart
• Facilitate communication by using the task board on agile ceremonies
Task Board
ProductBacklog
Item (PBI)or
User Story
Task
Burndown chart: progress against sprint plan
ALM 2: Delivery (Development)
• Version Control: TFS VC and/or git repository
• Database projects and dacpac deployments
• Database Unit Test Framework: MS Test
• Link Tasks to Check-Ins/Commits:
Keep them right
• Set compulsory Check-In comments
• Require Check-In to be allocated to a task – visibility
• Consider Gated Check-Ins if the builds are quick enough
ALM 2: Delivery (Be Prepared!)
• Configure Deployment Pipeline as early as possible• Helps envisage problems early
• Helps in planning and development
• Involve Ops early to avoid disappointment
• Deploy to Test/CI environment as often as possible• ideally every check-in/commit
• at least daily as practical
ALM 3: Deploy
• Use Deployment system: Release Manager or Octopus Deploy
• Automate Deployments: harness the PowerShell!
• Automate Approval Worklow: get your system do the legwork
ALM 4: Operate – Bug tracking
• Use TFS for logging Bugs and system issues
• Full traceability issue to resolution
• Web Interface
Let’s talk deployment!
• Release Manager• Better integration with TFS
• Work Items Tracking!
• Works well with BI projects• One-liner Build of any BI project with devenv.com
• Octopus Deploy • Free for 5 users/5 projects/20 tentacles
• Excellent RedGate integration
• Many other tools : XLDeploy, Cruise Control.NET, CHEF, etc
?
TFS 2017 Deployment
Drop location DeploymentAgent
TFS VC / Build / VS
Developer
Tester
E n v i r o n m e n t s
CodeBuild/Release Definitions
p i p e l i n e
Test Manager
OperationalSupport
Octopus Deploy architecture
Deployment ServerBuild/CI Server+VS + SSDT
Projects
Source Control
NuGetPackage
E n v i r o n m e n t s
Developer Tester Ops/Release Man
tinyurl.com/octo-build
T e n t a c l e s
L i f e c y c l e
Number of Agents
• TFS:• No longer a license issue: have as many as you need
• Secure TLS 1.2 communication with TFS Application Server
• Build agent should be separate from TFS
• Deployment agent must see the targets (firewall)
• Octopus:• 20 included in the free edition
Don’t like agents? Go agentless!
• Release Manager (DSC) or XL Deploy
• Utilize Windows Remote Management
• Can be perceived as more risky
• Harder to configure correctly
• More fun (for die-hard coders)
Concurrent Pipelines: you what?
• It is a number of Concurrent Releases actually running (not waiting)
• Every TFS 2017 server comes with one
• MSFT Rule of thumb: 1 for 10 devs ☺
• Extra pipeline for every Enterprise MSDN subscription
• No need to buy pipelines in TFS 2015 ☺
• Loosely equivalent to a project in Octopus
Deployment Tools summary
• Use what is accepted/supported in the company – don’t re-invent
• Octopus Deploy• Simpler to install and get started
• Can be cheaper if nothing is currently in place (free for 5x5x20 teams)
• Not MS BI – friendly: workarounds required
• TFS Release management• More sophisticated
• Better support for BI deployments
• Easy integration with Work Items
• Use if you already have it!
Deployment pipeline: logical steps
Developer Checks In Code
• Gated check-in for protection
• Check In Triggers for CI
Build is
Queued
• Build template
Auto-Deploy
to CI/Test
• Release template (TFS)
• Project + Lifecycle (Octopus)
Approve and release to Prod
• Deployment workflows (TFS)
• Dashboard (Octopus)
Build Release
Build: Off with his head?
• Headless build• Custom coding to build BI project
• Additional support overhead
• Devenv.com• Same as by developer (Visual Studio)
• Used in Azure
• Beware: Use 16.3 version of SSDT-BI (Build 14.0.60812) or older
• “Prime” SSDT-BI to make sure it knows what to do with SSAS Tabular
SSIS Tabular Visual Studio property
Deployment: each to their own
• Database projects: .DacPac (PowerShell or SQLPackage.exe)
• Analysis services: .ASDatabase (PowerShell or AS Deployment Wizard)
• Integration Services: .ISPac (SQL API or ISDeploymentWizard.exe)
• MDS: Import/Export (MDS API or MDSModelDeploy.exe)
• Reporting Services: .rds, .rdl (SSRS API or RS.exe)
Deploying Databases: script v dacpac
• Script: changes are manually scripted by a developer/tool• Visibility of what is going to change
• Finer control over the process (it seems)
• Works better for simple changes/schemas
• Dacpac: database model deployment• Consistency: fully transactional
• Designed to protect data integrity
• Repeatable
• Free of side-effects
Deploying Databases: SqlPackage.exe
• Hidden Gem of SQL Server Data Tools
• Program Files (x86)\Microsoft SQL Server\120\DAC\bin
• Can use the same profile file as in Visual Studio with your dacpac
• Very easy to use:
sqlpackage.exe /action:publish
/sourcefile <dacpacFilename>
/TargetServerName <serverName>
/TargetDatabaseName <DBName>
/Profile <profileFilename>
Deploying Databases: PowerShell
• # Add the DLL
• Add-Type -path "C:\Program Files (x86)\Microsoft SQL
Server\120\DAC\bin\Microsoft.SqlServer.Dac.dll"
• # Create the connection string
• $d = New-Object Microsoft.SqlServer.Dac.DacServices
"server=(local)"
• # Deploy the dacpac
• $d.Deploy(“C:\Path\AW.dacpac”, “AWDW”, $true, $options)
• See full working script at dataunited.co.uk/octo-dacpac
Deploying Cubes: Analysis Services Database
• Microsoft.AnalysisServices.Deployment.exe "C:\...\AdventureWorksCube.asdatabase" /s
• The path must be absolute!
• Only full deployments (all changes) are supported this way. If partial deployment is required, use API, or BISM Normaliser for Tabular.
• Use DeploymentWizard GUI to configure “Input Files”:
Deploying Cubes: Analysis Services XMLA
• Invoke-ASCmd PowerShell cmdletInvoke-ASCmd
–InputFile:”C:\MyFolder\DiscoverConnections.xmla”
• All paths must be absolute!
• If you get “The term 'invoke-sqlcmd' is not recognized …”:• Install PowerShellExtensions for the version of SQL Server on targets
• Run Import-Module SQLASCMDLETS to make use of them (comes with SQL Server Client Tools)
Deploying ETL: SSIS
• Deployment artefact: .ispac file
• ISDeploymentWizard.exe
/Silent /SourceType:File /SourcePath:“C:\...\AdventureWorks.ispac" /DestinationServer:“localhost" /DestinationPath:"__DestinationPath__“
• Consider a separate database project for server-level components• Environments (remember to scripts when you create them)
• Agent Jobs
• Security, etc
Other deployment
• SSRS• Check out “Continuous Integration for Reporting Services”
by Nat Sundar on the last SQL Bits: http://tinyurl.com/SSRS-CI• Includes deploying, refreshing and testing SSRS reports with nUnit
• MDS: MDSModelDeploy.exe• Run as administrator and on the MDS server (another Agent)
• Careful with model-only deployments – always check the results!
• Everything else: use the Power in PowerShell!• Just let it use the latest .Net framework: search for OnlyUseLatestCLR
Deployment summary: PowerShell or CMD?
• Complex projects require PowerShell: to many moving parts
• Simpler projects are easier to do with CMD
• If you can use CMD – do it!
• Both require SQL Server Client Tools installed on the Deployment Agent
Demo of CI using TFS
• I wish I can cover both, but there is no time…