““Notes from the Field”Notes from the Field”

Lessons learned developing a Lessons learned developing a distributed .NET applicationdistributed .NET application

Presenter Presenter

Darryl PollockDarryl Pollock

Squirrel Consulting Pty LimitedSquirrel Consulting Pty Limited

Sydney SQL SIG 26 October 2004Sydney SQL SIG 26 October 2004

Darryl PollockDarryl Pollock Squirrel Consulting Pty LimitedSquirrel Consulting Pty Limited

Developer for 18 yearsDeveloper for 18 years DBA for 7 yearsDBA for 7 years Consultant for 5 years including :Consultant for 5 years including :

– Development/ SQL and .NET ArchitectureDevelopment/ SQL and .NET Architecture– ClusteringClustering– Performance TuningPerformance Tuning– Large DB implementations (up to 1 TB)Large DB implementations (up to 1 TB)

darryl@squirrelconsulting.com.audarryl@squirrelconsulting.com.au

AgendaAgenda

Background of the applicationBackground of the application ArchitectureArchitecture Data ModellingData Modelling ConcurrencyConcurrency Reporting Services IntegrationReporting Services Integration Pearls of Wisdom Pearls of Wisdom DemoDemo (there is a QA session – but would prefer (there is a QA session – but would prefer

Questions throughout!)Questions throughout!)

BackgroundBackground SME’s seem to have core applications based

around 1 or 2 servers

These applications are business-critical and These applications are business-critical and often need monitoringoften need monitoring

BUTBUT SME’s cannot afford nor wish to manage large SME’s cannot afford nor wish to manage large

infrastructure monitoring systemsinfrastructure monitoring systems

BackgroundBackground clients began asking for a monitoring system for clients began asking for a monitoring system for

their their business-critical systems systems

The industry was researched, products tested, The industry was researched, products tested, evaluated for a 3 month periodevaluated for a 3 month period

No single product was found that met the No single product was found that met the following criteria:following criteria:

Criteria Lightweight – would not create performance Lightweight – would not create performance

issuesissues

Platform independent from NT upwardsPlatform independent from NT upwards

Secure, yet firewall-independentSecure, yet firewall-independent

Was not overly complex to useWas not overly complex to use

Provided hierarchical security (portal)Provided hierarchical security (portal)

Provided long term (trend) reportingProvided long term (trend) reporting

Could work across multiple organisationsCould work across multiple organisations

Low Cost!Low Cost!

SQL ServerAgent (RSM)

Lightweight/Platform IndependentLightweight/Platform Independent

SOAP/XML(Secure)

Licensing

Agent Monitoring

Performance/Events

Pre-configured Portal Technology (Simple)

Configuration(XML File)

(Simple/Forms Based Configuration)(Simple/Forms Based Configuration)

SOAP/XML

SOAP/XML

SOAP/XML

Reporting

SQL Server

4 Languages 4 Languages

Multiple Technologies/Multiple PlatformsMultiple Technologies/Multiple Platforms

Multiple applications competing for the same Multiple applications competing for the same resource:resource:– One SQL Server!One SQL Server!

The Challenge!The Challenge!

Building the Data Model Building the Data Model

Historical Data

Initially just _Total

tblCompanytblRSM

tblEvent

tblRSMViewPanetblRSMViewCounter

tblPerformanceObject

tblPerformanceCounter

tblCounterInstance

Users

tblCounterHistory

Building the Data Model - TestingBuilding the Data Model - Testing

Initially this table would grow by 100-300 rows every interval (1-3 minutes)

Performance of course was spectacular even with 15-20 agents running

Aggregation of the data was performed at insertion by way of a trigger (avg,max,min)

tblCounterHistory

Building the Data Model - TuningBuilding the Data Model - Tuning

tblCounterInstance

Reaching 1 million rows occurred with one agent running in a week

Performance of course was abysmal even with 2 agents running

Aggregation of the data was changed to occur hourly – and there was no archiving strategy in place – we’d encountered our first major issue. Why? Because our archiving strategy was not part of the design!! (Big mistake)

Once the table started exceeding 1 million rows, we were in trouble! The database was straining to view the data even with reasonable indexing

300 Rows became 1000 rows by allowing ALL

instances!

tblCounterHistory

Building the Data Model – Re-factoringBuilding the Data Model – Re-factoringtblCounterHistory

ID

LastDateTimePerformanceCounterID

We not only adjusted the model, but we built the first foundation of the report architecture

tblCounterView

ID

PerformanceCounterIDCounterInstanceIDRSMIDActive

tblCounterHistoryArchive

ID

RSMIDCounterInstanceIDValueLastDateTimePerformanceCounterID

tblCounterHistoryDaily

RSMIDCompanyIDMaxValueMinValueAvgValuePerfCounterPerfObjectCounterInstanceAggregateDate

Trigger

Deletion and Aggregation became hourly – only 1 hour of data was maintained

Deletion and Aggregation nightly

In the final product, we removed aggregation from the DB completely and moved to the agent – it took 7 weeks to get the agent to run efficiently – but worth it!

This became the “snapshot” of the current state

The Pyramid of ErrorThe Pyramid of Error

Flaw in Data Model

Re-Engineering of Database

Recoding of Web Services

Recoding of SQL Agent

Recoding of Portal

Concurrency – the cowards way out?Concurrency – the cowards way out? We weren’t building a large enterprise system, but a system that an SME We weren’t building a large enterprise system, but a system that an SME

would be running – one server to run probably everything…Which meant :would be running – one server to run probably everything…Which meant :

Most likely scenario was portal retrieving transactional data from a highly Most likely scenario was portal retrieving transactional data from a highly transactional system. (Does this sound like a disaster in the making?)transactional system. (Does this sound like a disaster in the making?)

Transactional data that was being sent to the presentation layer at the portal Transactional data that was being sent to the presentation layer at the portal experienced MAJOR concurrency issues. The only solution that would work for experienced MAJOR concurrency issues. The only solution that would work for us was :us was :

SHOCK! HORROR!!! Select with (nolock) SHOCK! HORROR!!! Select with (nolock)

Our alerts were real-time, so there was no perceived delay – it was a choice Our alerts were real-time, so there was no perceived delay – it was a choice that worked for that worked for this this application.application.

However this did not solve all our concurrency issues so….However this did not solve all our concurrency issues so….

Concurrency – letting SQL do its thing…Concurrency – letting SQL do its thing… Connection re-use – as soon as we adjusted the Web Service to have each Connection re-use – as soon as we adjusted the Web Service to have each

WebMethod use a single connection, our performance increased by about 20% WebMethod use a single connection, our performance increased by about 20% - this can be achieved either through code or connection pooling.- this can be achieved either through code or connection pooling.

This resulted in locks being held for shorter periods of timeThis resulted in locks being held for shorter periods of time

We replaced DataAdaptors with SQLCommand.ExecuteScalar wherever We replaced DataAdaptors with SQLCommand.ExecuteScalar wherever possible, thereby reducing the amount of data we were retrievingpossible, thereby reducing the amount of data we were retrieving

We adjusted all highly transactional, tables to use Row-Level locking, but left We adjusted all highly transactional, tables to use Row-Level locking, but left the others alone.the others alone.

We stopped specifying any type of lock to hold in our procs.We stopped specifying any type of lock to hold in our procs.

We then sat back and testedWe then sat back and tested

SQL held locks, but released them, fast! – This was the result we wanted.SQL held locks, but released them, fast! – This was the result we wanted.

Large Data Sets and what to do with themLarge Data Sets and what to do with them Retrieving a large number of events from the database became a performance Retrieving a large number of events from the database became a performance

bottleneck – instead of just relying on tuning – we not only tuned the indexes bottleneck – instead of just relying on tuning – we not only tuned the indexes on the event table but used .NET’s built in DataView and DataFilter objects.on the event table but used .NET’s built in DataView and DataFilter objects.

The proc retrieved the entire dataset from a well-indexed table, and instead of The proc retrieved the entire dataset from a well-indexed table, and instead of going back to the database and retrieving a filtered new rowset – we passed going back to the database and retrieving a filtered new rowset – we passed the original DataSet into a DataView and applied a filter – with excellent resultsthe original DataSet into a DataView and applied a filter – with excellent results

Eeek!!!!!!! Dot net Code!!! (This little snippet increased performance by 100%!)Eeek!!!!!!! Dot net Code!!! (This little snippet increased performance by 100%!)

If Me.DataFilter <> "" ThenIf Me.DataFilter <> "" Then

Dim oDV As DataView = Me.RSMCounters.DefaultViewDim oDV As DataView = Me.RSMCounters.DefaultView

oDV.RowFilter = Me.DataFilteroDV.RowFilter = Me.DataFilter

Me.datagridRSMCounters.DataSource = oDVMe.datagridRSMCounters.DataSource = oDV

Me.datagridRSMCounters.DataBind()Me.datagridRSMCounters.DataBind()

ElseElse

Me.datagridRSMCounters.DataBind()Me.datagridRSMCounters.DataBind()

End IfEnd If

Reporting Services IntegrationReporting Services Integration

tblReports

tblReportEnvironmentRetrieve List of Reports

Select Report

Get Server Name

Build URL Based on Parameters of where we were in the portal

Redirect Browser to URL

tblReports

Collaborationor

(How not to be bullied by developers!) In most cases, you want your stored procedure to In most cases, you want your stored procedure to

represent your logical model …howeverrepresent your logical model …however

Sometimes it is best to let the application do the logic Sometimes it is best to let the application do the logic for retrieving the data structure vs. retrieving it all in one for retrieving the data structure vs. retrieving it all in one “hit” in a complicated proc“hit” in a complicated proc– why? because it is quicker!why? because it is quicker!

For example:For example:

Our database – the philosophyOur database – the philosophy

Golden RulesGolden Rulesdon’t put app logic in dbdon’t put app logic in dbdon’t put db logic in appdon’t put db logic in app

Use your db as a data storage device, nothing Use your db as a data storage device, nothing moremore

we only use a few cursors just for initialisation – we only use a few cursors just for initialisation – and they are run once. If you think you need a and they are run once. If you think you need a cursor to run more than once – forget Golden cursor to run more than once – forget Golden Rule #1!Rule #1!

use .NET to loop through a data set and evaluate use .NET to loop through a data set and evaluate each row..its quick and far more efficient each row..its quick and far more efficient

DRIDRI

If you choose to have your application manage the If you choose to have your application manage the referential integrity of your application, then the referential integrity of your application, then the integrity is as strong as your worst developer.integrity is as strong as your worst developer.

The database is never wrong! The database is never wrong! We let SQL generate the error and then handle it We let SQL generate the error and then handle it

at the application layer. at the application layer. The application should be ignorant of the physical The application should be ignorant of the physical

database, and only be concerned with how to read database, and only be concerned with how to read and write objects.and write objects.

Use the Cascading deletes and updates – it can Use the Cascading deletes and updates – it can save hours of development time. save hours of development time.

The FutureThe Future

2 Way SOAP communication – creating 2 Way SOAP communication – creating SQL ‘queues’ of commands at the agentSQL ‘queues’ of commands at the agent

Wireless job summary and control agentsWireless job summary and control agents Yukon – execute all functionality via CLRYukon – execute all functionality via CLR Extending the architecture beyond Extending the architecture beyond

monitoringmonitoring– Retail sales summaries Retail sales summaries – GPS and SOAP GPS and SOAP

ContactContact More Info - More Info - www.squirrelconsulting.com.au/LearnMore.htmwww.squirrelconsulting.com.au/LearnMore.htm

Demo – Demo – www.remotesquirrel.comwww.remotesquirrel.com

Company Company www.squirrelconsulting.com.auwww.squirrelconsulting.com.au

darryl@squirrelconsulting.com.audarryl@squirrelconsulting.com.au