DAT336 Connected vs Disconnected Data Access in ADO.NET

DAT336 Connected vs Disconnected Data Access in ADO.NET

Pablo CastroPablo CastroProgram Manager – ADO.NET TeamProgram Manager – ADO.NET TeamMicrosoft CorporationMicrosoft Corporation

AgendaAgenda

Disconnected is goodDisconnected is goodSome connected bits can helpSome connected bits can help

Balancing connected/disconnectedBalancing connected/disconnectedScrolling and streamingScrolling and streaming

Custom aggregationCustom aggregation

Incremental loadIncremental load

SummarySummary

Disconnected?Disconnected?

In the context of this talk:In the context of this talk:““Disconnected” apps are online Disconnected” apps are online applications that don’t keep database applications that don’t keep database connections open for long periods of time connections open for long periods of time such as the lifetime of a sessionsuch as the lifetime of a session

Usually 3-tier applicationsUsually 3-tier applications

We’re not going to discuss “off-line” We’re not going to discuss “off-line” applications where a user can operate an applications where a user can operate an application even if the server is not application even if the server is not availableavailable

Disconnected is GoodDisconnected is Good

Simpler to designSimpler to design3-tier applications follow the 3-tier applications follow the disconnected model naturallydisconnected model naturally

Simpler to code once the pieces are setSimpler to code once the pieces are seti.e. data binding, marshalling in 3-tier i.e. data binding, marshalling in 3-tier apps, no connection management for apps, no connection management for long-lived business objectslong-lived business objects

Simpler to make apps scaleSimpler to make apps scaleMiddle-tier layer easy to scale-outMiddle-tier layer easy to scale-out

It’s harder to scale-up/out database serversIt’s harder to scale-up/out database servers

Disconnected ADO.NETDisconnected ADO.NET

ADO.NET has first-class disconnected ADO.NET has first-class disconnected apps supportapps support

DataSet as a relational data cacheDataSet as a relational data cachePlays well with remoting & web servicesPlays well with remoting & web services

Single implementation, no database-specific Single implementation, no database-specific behaviorbehavior

DataAdapter/DataSet provide services for DataAdapter/DataSet provide services for getting, updating and merging datagetting, updating and merging data

Custom types for non-relationalCustom types for non-relational

…but too disconnected……but too disconnected…

Handling large volumes of dataHandling large volumes of dataBatch processingBatch processing

ReportingReporting

Custom aggregationCustom aggregation

Performance-sensitive code-pathsPerformance-sensitive code-pathsASP.NET pages in apps without heavy ASP.NET pages in apps without heavy business logicbusiness logic

Gateway applicationsGateway applications

Avoid buffering large Avoid buffering large portions of dataportions of data

Avoid data-shipping costsAvoid data-shipping costs

Discrete Objects ScenariosDiscrete Objects Scenarios

This is the “easy part”This is the “easy part”Stand-alone business objectsStand-alone business objects

Business object state representationBusiness object state representationDataSets – ADO.NET helps with retrieval, DataSets – ADO.NET helps with retrieval, update and change trackingupdate and change tracking

Custom objects – model your business Custom objects – model your business entity state using regular classesentity state using regular classes

Remoting or webservices enable this Remoting or webservices enable this scenario easilyscenario easily

Sessions in the Middle TierScenariosSessions in the Middle TierScenarios

Per-session state can’t be avoided Per-session state can’t be avoided sometimessometimes

If needed, keep only in-memory stateIf needed, keep only in-memory state

No database connections if at all possibleNo database connections if at all possible

Quite common in ASP/ASP.NET appsQuite common in ASP/ASP.NET apps

““Sticky sessions” take care of scale-outSticky sessions” take care of scale-outSo it’s not that bad if you don’t keep So it’s not that bad if you don’t keep connections or other external resourcesconnections or other external resources

Scrolling and StreamingScenariosScrolling and StreamingScenarios

In general, handling large results in In general, handling large results in piecespieces

UI: scrolling, pagingUI: scrolling, paging

Batch processing: chunking, scanning Batch processing: chunking, scanning large resultslarge results

Custom aggregationCustom aggregation

Options change depending on each Options change depending on each casecase

Scrolling, PagingScrolling, Paging

Goal is to fetch rows from a large Goal is to fetch rows from a large result, a few at a timeresult, a few at a time

CursorsCursors

DataAdapter.Fill methodDataAdapter.Fill method

SQL-based solutionsSQL-based solutions

Scrolling, Paging: CursorsScrolling, Paging: Cursors

Not available in all databasesNot available in all databases

Provide scrolling supportProvide scrolling supportSometimes even bi-directionalSometimes even bi-directional

No need for extra logic in the applicationNo need for extra logic in the application

Scalability issuesScalability issuesRequire to maintain stateRequire to maintain state

Database connections, keyset or temporary Database connections, keyset or temporary tables in the databasetables in the database

Cursor escalationCursor escalationServer may need to materialize some/all dataServer may need to materialize some/all data

Scrolling, Paging: CursorsScrolling, Paging: Cursors

Design issuesDesign issuesHard to include in 3-tier applicationsHard to include in 3-tier applications

Need to keep connection/cursor objects Need to keep connection/cursor objects alive in middle tieralive in middle tier

How is data propagated to the How is data propagated to the presentation layer?presentation layer?

Tends to be a chatty interfaceTends to be a chatty interface

Result stabilityResult stabilityNeed to use transactions or static cursors Need to use transactions or static cursors if stability is requiredif stability is required

Scrolling, Paging: Fill()Scrolling, Paging: Fill()

ADO.NET DataAdapter.Fill() methodADO.NET DataAdapter.Fill() methodThere’s an overload that takes first row There’s an overload that takes first row and number of rowsand number of rows

Under the covers, this method:Under the covers, this method:Skips rows as neededSkips rows as needed

Copies as many rows as requested to the Copies as many rows as requested to the target DataTabletarget DataTable

Scans and discards the rest of the rowsScans and discards the rest of the rows

Don’t use it for paging in large Don’t use it for paging in large result-setsresult-sets

Scrolling, Paging: SQLScrolling, Paging: SQL

If at all possible, use SQL constructs If at all possible, use SQL constructs for pagingfor paging

Stored-procedures Stored-procedures if you know the table schema and sort orderif you know the table schema and sort order

SQL can help in other cases SQL can help in other cases where the query is not known but constrainedwhere the query is not known but constrained

Scalability/performance issuesScalability/performance issuesTime taken to execute query is not Time taken to execute query is not amortized across requestsamortized across requests

Scrolling, Paging: SQLScrolling, Paging: SQL

Design issuesDesign issuesFits nicely for 3-tier appsFits nicely for 3-tier apps

Each page is an independent database Each page is an independent database operationoperation

No state held between hitsNo state held between hits

Ad-hoc queries are hard to handleAd-hoc queries are hard to handle

Result stabilityResult stabilityNeed to use transactions if stability is Need to use transactions if stability is requiredrequired

StreamingStreaming

Goal is to handle very large result-setsGoal is to handle very large result-setsNo buffering proportional to size of dataNo buffering proportional to size of data

Concurrency issuesConcurrency issues

InterleavingInterleaving

Common scenarios for streamingCommon scenarios for streamingBatch processingBatch processing

ReportingReporting

Custom aggregationCustom aggregation

Streaming: DataReadersStreaming: DataReaders

ADO.NET providers have an streaming ADO.NET providers have an streaming interfaceinterface

DataReader class exposes a row at a timeDataReader class exposes a row at a timeSome minor buffering might happen internallySome minor buffering might happen internally

Can scan millions of rows without taking Can scan millions of rows without taking much resourcesmuch resources

Scalability issuesScalability issuesUsually large scans happen in batch Usually large scans happen in batch processesprocesses

Not many at the same timeNot many at the same time

Streaming: DataReadersStreaming: DataReaders

Design issuesDesign issuesDataReaders cannot be marshaled across DataReaders cannot be marshaled across tierstiers

Move the batch process code to the middle Move the batch process code to the middle tiertier

Send the data in chunks to the next tier (too Send the data in chunks to the next tier (too much overhead in most cases)much overhead in most cases)

Contention can be highContention can be high

Result stabilityResult stabilityDepends on the isolation levelDepends on the isolation level

Custom Aggregate LogicCustom Aggregate Logic

Similar case: scan lots of rowsSimilar case: scan lots of rowsBut end-result is a small result-setBut end-result is a small result-set

No need to ship the data out of the No need to ship the data out of the serverserver

Cursors can help hereCursors can help here

Scan and aggregate inside the serverScan and aggregate inside the server

Ship only the aggregated information to Ship only the aggregated information to the clientthe client

Avoids moving lots of data across tiersAvoids moving lots of data across tiers

Incremental LoadScenariosIncremental LoadScenarios

Present first bit of data in UI quickPresent first bit of data in UI quick

Incrementally load the rest in the Incrementally load the rest in the backgroundbackground

DataSet merge support is great hereDataSet merge support is great here

Be aware of multi-threading issuesBe aware of multi-threading issues

Incremental Load & MergeIncremental Load & Merge

ADO.NET DataSet can merge resultsADO.NET DataSet can merge results

Incremental load UIIncremental load UIChunking API in the middle tierChunking API in the middle tier

Bring down the first DataSet and display itBring down the first DataSet and display it

As more data comes, merge the DataSets As more data comes, merge the DataSets and update UIand update UI

This even preserves changes in existing dataThis even preserves changes in existing data

Multi-threading issuesMulti-threading issuesDataSet is not thread-safeDataSet is not thread-safe

Same for WinForms UI controlsSame for WinForms UI controls

SummarySummary

Disconnected is goodDisconnected is goodADO.NET has great support for itADO.NET has great support for it

Good to start disconnected by defaultGood to start disconnected by default

You will need some connected piecesYou will need some connected piecesADO.NET also helps thereADO.NET also helps there

You can add connected parts as neededYou can add connected parts as needed

Extremes can hurt your app Extremes can hurt your app performance or scalabilityperformance or scalability

Attend a free chat or web castAttend a free chat or web casthttp://www.microsoft.com/communities/chats/default.mspxhttp://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asphttp://www.microsoft.com/usa/webcasts/default.asp

List of newsgroupsList of newsgroupshttp://communities2.microsoft.com/http://communities2.microsoft.com/communities/newsgroups/en-us/default.aspxcommunities/newsgroups/en-us/default.aspx

MS Community SitesMS Community Siteshttp://www.microsoft.com/communities/default.mspxhttp://www.microsoft.com/communities/default.mspx

Locate Local User GroupsLocate Local User Groupshttp://www.microsoft.com/communities/usergroups/default.mspxhttp://www.microsoft.com/communities/usergroups/default.mspx

Community sitesCommunity siteshttp://www.microsoft.com/communities/related/default.mspxhttp://www.microsoft.com/communities/related/default.mspx

© 2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.