Deep Dive into Data Management in SharePoint applications

Raj Chaudhuri

Quasi-statutory warning• You are going to hear a lot of opinion

in this session. My opinion.

Agenda• The SharePoint data modelS (note the capital S)• The core SharePoint “Lists” data model : our

focus today• Discussion of:

– Data definition – Data manipulation– Querying and extraction

• The Object Model: the SharePoint “business tier”

• Conclusion

THE SHAREPOINT DATA MODEL(S)

SharePoint Data Model(S)• Data stored in external data stores,

accessed directly using custom components• Data stored in external data stores,

accessed using BCS or Excel Services• Data stored in SharePoint “service

databases”, accessed using Service Applications

• Data stored in Web Part Properties (content DB), accessed using the web part manager

• Data stored in SharePoint Lists

THE SHAREPOINT “LISTS” DATA MODEL

Characteristics• Designed for collaborative applications,

and as such:– Updates are more important than Inserts– Both are more important than querying–Multiple people are expected to work on

the same instance of data at the same time

– So, no locks. Instead, auditing and versioning. Caveat: Check in/check out model in document libraries

More Characteristics• Designed for user serviceable applications. As

such:– The data structures can change at any time

• Hence, each data structure is largely self-contained. Relations in the classical sense are possible, but cludgy

– So can relations• Hence, rather than consider relationships between

entity types, we should consider relationships between entity instances

– The UI must automatically adapt to match• So, STOP asking for “customized” UI for SharePoint data

entry. You’re missing the POINT of SharePoint.

Even More Characteristics

• Permissions are an important part of the data model

• As are associated actions: reactive ones like “triggers” (Event Receivers), and proactive ones like associated custom actions, quick steps etc.

Logical Data ModelLogical Concept SharePoint “List” data model concept

Data Type Field Type. This determines how the data is stored in the underlying data store, how it is represented in user interfaces, and uniquely in SharePoint, the actual user interface used to display and edit this type of data. More types can be created by developers

Attribute Column. Columns in SharePoint could be local to a collection of data, or reusable across data collections. They also contain metadata that directly affects the user interface.

Tuple (a collection of columns representing one unit of data)

List Item. A List Item can also contain associations. For instance, list items can have attachments or themselves represent files, can exist inside “folders”, can have multiple versions, can go through a workflow etc. Ultimately, all this is represented through columns.

Table (a row-ordered collection of tuples)

List. A list is not just a collection of list items. It is also a way for relating items together, a link between items and various user interfaces, and a provider of context to item data.

Conceptual Data ModelConceptual (uh) Concept SharePoint Concept

Attribute Column, as explained in the previous slide.

Entity Content Type, and not list. Every list item in every list is an instance of a content type. Like a list, a content type can provide context to data. For example, only items of a particular content type may be associated with certain UI actions, events or workflows.

Entity Set List. A single list can have items of different content types.

Columns• Columns are identified by a GUID and an internal

name, both of which need to be unique in the column’s namespace. They also have a Display Name, which need not be unique, can be changed by the user, and is usually localized in Language Packs.

• Columns can be defined at the list level. They are then available in that list only. Such columns are called List Columns.

• Columns can also be defined at the site level. They are then available on that site and sites below it in the site collection. Such columns are called Site Columns.

Columns Contd.• Site columns can be added to lists. A particular

site column may be added to a list only once.• When a site column is added to a list, the list

actually gets a column which is an exact copy of the site column, not the site column itself. This column is called a site column in list. Its properties may be changed in the list.

• A site column in list maintains a link to the original site column. Any changes made to the site column can be applied to all site columns in lists that were created from that site column.

Content Types• Content Types in SharePoint can, and in fact have

to, derive from other content types. A derived content type can inherit all fields of the base content type.

• All content types in SharePoint ultimately derive from a base content type called Item. An entire category of content types derive from another type called File. These represent files in library-type lists. File itself derives from Item.

• Content Types are identified by a unique ID. This ID not only identifies the content type, but also identifies the base content type.

Content Types Contd.• Content Types are defined at the site level.

They are then available on that site and sites below it in the site collection.

• Content Types can also be defined at the list level. They are then available only in that list.

• A site content type can be associated with a list. This creates an exact copy of the site content type in the list, called a Content Type In List. This maintains a link to the original, and any changes made to the original can flow down to all copies.

Lists• Lists are created from List Definitions or List

Templates. A definition defines what content types a list can have, and what columns.

• The columns in a list can come from the content types, or be new ones defined in the list itself, or be site columns in the list.

• The number of columns that the list receives from its associated content types is the union of the columns in all those content types.

Lists• There are a lot more columns in a list

than are apparent. For example, each workflow associated with a list causes a column to be maintained. There is a column that tracks the content type of each item, another which tracks the version, a column called FsObj that tracks if the item is a file or folder, and so on.

Physical data model• This should not matter. It is meant to be

opaque to all consumers of SharePoint technology.

• That means, do NOT directly modify, query, or even look at the structures of, any SharePoint database.

• NOT EVEN if some blog of some expert tells you to.

• NOT EVEN if I tell you to• Do NOT, okay?

That said…• List items are stored in real SQL Server

tables, as real rows. • A single SharePoint list item can be

wrapped over a number of SQL Server rows.

• The number is 6 by default. It can be adjusted, but not through any UI; only programmatically. Do not do this. Unless you know what you are doing. And you don’t. I don’t either. Not yet.

We really should not be discussing this…

• The total number of bytes that a single SharePoint list item can occupy is hardcoded to 8000.

• This total is to be reached by adding the number of bytes of storage required by each column. That, in turn, depends on the column’s field type. This does not include files and attachments.

• 256 bytes out of 8000 are reserved for system-defined columns.

• For more details, read:– http://technet.microsoft.com/hi-in/library/cc262787(en-

us).aspx (SharePoint Server 2010 capacity management: Software boundaries and limits)

One more time• Worry about the physical only as a

last resort• And DO NOT look inside the

databases.

DATA DEFINITIONDiscuss:

DATA MANIPULATIONDiscuss:

THE OBJECT MODEL: SHAREPOINT’S “BUSINESS TIER”

The object model• ALL access to SharePoint data (at

least for manipulation) must happen via the object model.

• This ensures the maintaining of context, security and integrity.

• This also causes event receivers to fire, and workflows to execute as appropriate.

Object model hosts• The object model may be hosted by– A SharePoint web application– A SharePoint service application– A SharePoint Windows service, especially

the Timer service– A custom application (Careful here. An

application that uses the SharePoint web services is indirectly using the OM hosted in a SharePoint web application. Ditto an app that uses the Client Object Model).

Effect• The object model, and hence the

host, directly talks to the appropriate databases.

• Event receivers are executed in the host. Workflows execute in the host.

QUERYING AND EXTRACTION

Discuss

In Conclusion• The SharePoint “Lists” data model is a very powerful and

flexible model, IF…– Your data is collaborative in nature, and– User-servicability is important to you

• Do NOT try to use lists instead of SQL database tables. They are meant for different kinds of usage. For example:– Do not use lists if your data is insert-heavy, updated rarely,

and queried frequently.– Especially the last. If you need complicated reporting, you

should not be using lists.– At least, be prepared to move the data from lists into some

other data store before you do heavy-duty reporting. Just like we do for data warehouses.

In Conclusion• And remember, lists are a package deal. You

get centralized backup, user servicability, a simplified permission system, workflows, event receivers, and an adaptive user interface.

• You cannot pick and choose. You get the whole deal if you decide on lists.

• And so, just as an example, please stop trying to precisely customize list forms. You are defeating the entire POINT…of SharePoint

Raj Chaudhuriraj@rajware.netraj.chaudhuri@communitiesrus.in +919930217314

Thank You