23
SSIS Field Notes Darren Green Konesans Ltd

SSIS Field Notes

  • Upload
    ravi

  • View
    33

  • Download
    0

Embed Size (px)

DESCRIPTION

SSIS Field Notes. Darren Green Konesans Ltd. My Checklist. Standards Logging Configuration. Common Princiapls. Common design decisions or patterns Logging Frameworks Custom vs Stock. Basic Standards. Solution and Project structure ETL vs ELT Staging Custom components. - PowerPoint PPT Presentation

Citation preview

Page 1: SSIS Field Notes

SSIS Field Notes

Darren GreenKonesans Ltd

Page 2: SSIS Field Notes

My Checklist

• Standards• Logging• Configuration

Page 3: SSIS Field Notes

Common Princiapls

• Common design decisions or patterns• Logging• Frameworks• Custom vs Stock

Page 4: SSIS Field Notes

Basic Standards

• Solution and Project structure• ETL vs ELT• Staging• Custom components

Page 5: SSIS Field Notes

Source Control

• Products– Team Foundation Server– Subversion (Visual SVN)– Others…

• Issues– Cannot merge or use standard conflict resolution

• BIDS Helper- Smart Diff– TFS, SourceSafe,File

• BI Smart Diff– Subversion

Page 6: SSIS Field Notes

Naming Conventions

• Prefix notation, e.g. DFT for Data Flow Task– http://consultingblogs.emc.com/jamiethomson/

• Expand Name property– SQL Create Year Staging Table

• Expand Description property– Create the year named staging table. Any existing

table will be dropped first.• Documentation tools are not clever

Page 7: SSIS Field Notes

Logging

• Performance monitoring– Real-time monitoring– Trending to justify upgrades• Re-write problem packages• Upgrade hardware / network / environment

• Problem solving– Why did the job fail last night?

Page 8: SSIS Field Notes

Logging Options

• Built in SSIS logging– Good for standard stuff, including errors

• Maintenance routine to prune records• Delete Info daily, Warnings weekly, Errors monthly

• Custom SSIS logging– Log process specific metrics - row counts– Event Handler or Control Flow

• Windows Logging– Event Log– Performance Monitor

Page 9: SSIS Field Notes

Prune – Delete in ChunksSET @Count = 1000WHILE @Count IS NOT NULL AND @Count > 0BEGIN DELETE TOP (@Count) FROM dbo.sysdtslog90 WHERE StartTime < @MinStartTime SET @Count = @@ROWCOUNT

-- Pause for 0.2 seconds WAITFOR DELAY '000:00:00.200'END

Page 10: SSIS Field Notes

Frameworks

• Consistent logging approach• Process or package state– Passing state between package processes– Saving state for the next run• Last extract date• Complete run only once per day

• Dynamic execution workflow– Managed in tables not Control Flow• Easier to manage logical units and reuse

Page 11: SSIS Field Notes

Frameworks

• Standard approach reduces costs– Lower support costs– Higher quality through reuse

• Cost of applying and maintaining framework– Maintainable frameworks are key!– Custom components encapsulate code– Use the API to bulk apply changes

Page 12: SSIS Field Notes

Custom Components

• Task• Pipeline Component– Source– Destination– Transformation

• Log Provider• Connection Manager• For Each Enumerators

Page 13: SSIS Field Notes

Using Custom Components?

• Easy to manage and update– Good for re-use– Can add/edit functionality easily – One file per machine for all packages– Good for frameworks or complex operations– Good debugging and testing support

• Require .NET development skills• External dependency– Additional step during the initial deployment, but also

single update step thereafter

Page 14: SSIS Field Notes

Stock Components & Scripting

• Faster to develop• Familiar and easy to understand– Don’t write your own data flow engine in a script

component!– Acknowledge the need for reuse when it exists and

create a shared external assembly • Self contained - No external dependency• Copy and paste package maintenance– Not good for frameworks or common patterns

Page 15: SSIS Field Notes

Recovery & Restarts

• Checkpoints – Native• CheckpointFileName, SaveCheckpoints,

FailPackageOnFailure

– Task level restart only– Partition your Data Flow• Raw files

– Variable values are persisted– Configurations not refreshed– Event handlers within checkpoint scope

Page 16: SSIS Field Notes

Recovery & Restarts

• Auto-Recovery – Roll your own– Check with IF EXISTS…– Framework workflow• Table of packages with status• Package or task level restart

– Variables and precedence constrains expressions

– Delete and re-load• No change tracking or updates

Page 17: SSIS Field Notes

Wake up – a quick demo…

Page 18: SSIS Field Notes

Anti-Pattern #1

• Send Mail Task for Errors– Control Flow or Event Handlers– Exceptions cause multiple events– Use external host to handle exceptions• SQL Agent

– Use a report

Page 19: SSIS Field Notes

Anti-Pattern # 2

• Multiple Configuration Types– Use one configuration type• Use indirect or ordered pointer configuration

– Easier for support and deployment

Page 20: SSIS Field Notes

Sorting

• Why sort?– Merge Join

– Ordered Analysis• Running totals• Ranking• Script Component• Custom Component • Rank Transform

Page 21: SSIS Field Notes

Sorting

• How to sort?– Sort Transformation– Ordinal Nsort– Ivolva ExtraSort (2005)– Sort in Source – ORDER BY

Page 22: SSIS Field Notes

Sorting

Page 23: SSIS Field Notes

Thank you

• Questions?

• Darren Green– [email protected]– www.konesans.com