Embed Size (px)
DESCRIPTION
saddsa
Citation preview
www.aditi.com
Abstract Significant work on the column store has changed the storage paradigm for data warehousing. The column store supported by vector based query execution and substantial progress in data compression have edged the technology as potential game changers. Microsoft targeting high on next-gen technologies increasing use of in-memory but memory optimized techniques in
SQL 2012. One of such most expectant implementation is xVelocity Column Store Indexes. This paper contains the detailed discussion on column store touching on the basics, limitations & design considerations with the demonstration examples. Columnar and Column store will be used interchangeably
in the below discussion.
xVelocity Columnstore Indexes
VertiPaq-ing With SQL 2012
www.aditi.com
Audiences This paper targets IT Planners, Architects, BI Users, CTO’s and CIO’s evaluating the SQL 2012 answering their large data grounded business needs. It also targets the enthusiasts of 2012 to provide new dimensions and out of box thinking to the organization to maintain data using SQL 2012.
Overview Data is growing exponentially and performance is becoming a recurring cost for the organizations. Performance impact can be broadly factorized as (1) I/O based operations (2) Memory based operations & (3) Operations to transfer data in N/W or other peripherals. I/O
and Memory are highly dependent on the method of storage and querying. Queries can be seen as (1) read-only workloads which are mostly reporting and DW systems and (2) the read-write workloads mostly OLTP systems. The potential game changer in read-only workloads is the storage method to minimize I/O and Memory based operations where-as conventional RDBMS stores data in row based storage system. Based on the columnar design a gain or speedup in queries can be seen from 10X up to 1000X.
For instance suppose the employee information looks like … ID Name Street Address
32498 Diamond John Crouse Manson
45298 Mary Anglos Wilson Street
RDBMS Relational Data Base Management System HCC Hybrid Columnar Compression
I/O Input Output FCC Full Columnar Compression
N/W Network LOB Large Objects
OLTP Online Transaction Processing OLAP Online Analytical Processing
DW Data Warehouse CSI Column Store Indexes
ETL Extract Transform & Load
Acronyms
www.aditi.com
Then in the conventional RDBMS (e.g. SQL Server 2008) it will be stored in row by row-wise fashion as shown in the Diagram-1 below.
Diagram-1
Where-as the Columnar Storage (e.g. SQL Server 2012 using column store index) will store it columnar fashion as shown in Diagram-2 below.
Diagram-2
Why Columnar IndexesThere is a great debate for the columnar structure. Below are benefits of using columnar indexes specifically to SQL Server 2012
Thought is to start the result driven discussion. Below are the graphs for the query performance results. We have started with the 12.5 million rows and doubled it every time till 400 million records to get total sales across products.
The comparison of query time with “Column Store Index” Vs. “Conventional Indexes” are exceptionally revealing in favor of column store indexes. The graph 1 and graph 2 shows the big gap of hundreds of seconds. The result shows
that warm cache is taking very less time comparatively. The graph-3 below is gain in “X” number of times in warm and cold cache. The result really excites the use of CSI.
Astonishing Results
VertiPaq-ing with xVilocity Columnnar Indexes
www.aditi.com
12.5 25 50 100 200 4000
200
400
600
800
1000
1200
1400Query Execution - Cold Cache
Column Store Indexes
Conventional Indexes
Number of Rows (in millions)
Que
ry E
xec T
ime
(in se
c.)
12.5 25 50 100 200 4000
200
400
600
800
1000
1200
1400Query Execution - Warm Cache
Column Store IndexesConventional Indexes
Number of Rows (in millions)
Que
ry E
xec T
ime
(in se
c.)
12.5 25 50 100 200 4000
15
30
45
60
75
90Gain in query performance - Warm Vs. Cold Cache
Warm Cache
Cold Cache
Number of Rows (in millions)
Perf
Gai
n (n
umbe
r of ti
mes
)
Graph-3
Graph-4
Graph-5
www.aditi.com
• SAVES THE STORAGE ON THE DISK BY MINIMIZING THE CREATION OF MULTIPLE INDEXES
AND SPECIALLY THE COVERING INDEX FOR QUERY PERFORMANCE.
• VERY HIGHLY COMPRESSED DATA AND EVEN MORE COMPRESSED BY HIGHER DEGREES
ON DUPLICATION.
• REDUCES DATA FOOTPRINT BY MINIMIZING THE USES OF PRE-AGGREGATED TABLES,
ACCUMULATED SNAPSHOT FACT TABLES AND MATERIALIZED VIEWS IN THE DATABASES.
• REDUCES COMPETENCY COST IT HAS THE CAPABILITY TO SCALE BEYOND SMALL
AND MEDIUM SIZE CUBES AND THUS REDUCE THE COST INCURRED IN ACQUIRING
CAPABILITY AND COST OF MAINTENANCE.
Faster Query Performance
• VERY HIGH DEGREE TO RELEVANT DATA READ ALMOST NEAR TO 99% IN MOST OF THE CASES.
ONLY NECESSARY COLUMNS TO ANSWER THE QUERY ARE READ AND AGGREGATED FROM THE
DISK. THIS POTENTIALLY REDUCES THE NUMBER OF READS AND SPEEDS UP THE QUERY.
• SPEEDS UP I/O BOUND QUERIES BY USING THE INDEX ONLY PLANS QUERIES THE INDEXES AND
REDUCES TO GO TO ROW TABLES RATHER.
Cost Saving
www.aditi.com
VartiPaq-ing & Appolo
VertiPaq-ing is vertical partition of the data or in other words storing in the data in column-wise fashion. The Diagram-3 shows the difference between the row and column store data layout in terms of pages which is basic unit of storage. For detailed discussion refer to the Basics Behind the Scenes section below. The goal behind it is to accelerate the common DW queries.Appolo is the code name available in SQL 2012 with the new feature available targeting
• FULL COLUMN STORE INDEXES WITH THE VERTIPAQ COMPRESSION ALGORITHM.
• THE COMPRESSION IS OPTIMIZED TO TAKE VERY LESS SPACE FOR THE DUPLICATE DATA IN
COLUMN MANNER BECAUSE OF THE SAME TYPE OF DATA (FOR EACH COLUMN) BEING STORED
TOGETHER.
• THE VECTOR BASED EXECUTION OF QUERY WHICH EXECUTES THE DATA IN BATCHES RATHER
THAN
ROW BY ROW OPERATION.
• MEMORY OPTIMIZED IN-MEMORY DATA; AN XVELOCITY FEATURE.
www.aditi.com
xVelocityxVelocity is term used by SQL Server family to define next-generation technologies. These technologies targets surprisingly high query performance in modern hardware. They are optimized
to use multi cores and high memory. Some more utilization of these techniques are there in Analysis Services and PowerPivot. Portions of data are moved in and out of memory based on the memory available in the
machine. Concisely they are highly optimized in-memory operations. Below is the screen shot taken during column store index creation of CPU utilization by xVelocity technologies.
Basics Behind the ScenesFull Column Store & Hybrid Column StoreSQL 2012 is full columnar storage where each column is compressed and stored together. This technique has its own advantages but it may negatively impact the performance on accessing more columns or perform small number of updates (although SQL 2012 has read-
only indexes). Refer to diagram-2 and 3 for details. On the other hand hybrid column used both rows and columns to store data. Hybrid technique creates column vector for each column, compresses and stores in data blocks. The compression unit contains more than one data blocks and it contains all columns for a row. The rows
spans over multiple data blocks. The diagram-5 shows the detail of the concept. This way the large amount of compression is achieved as well as the performance issues of the full columnar databases is also mitigated.
Graph-7
Graph-8
www.aditi.com
For the warehousing scenario the HCC approach many times is less performing because of
• WHILE DOING READ OPERATION IT WILL READ THE COMPRESSION UNIT, WHICH IS
COMBINATION OF THE SET OF ALL THE ROWS. SO IT WILL END UP READING ALMOST ALL THE
COLUMNS EVEN IF ONLY 20% OR 30% OF COLUMNS ARE NEEDED. MOST OF THE CASES THE
READS ARE INCREASED AS COMPARED TO FULL COLUMNAR APPROACH.
• HCC REDUCES THE MAIN MEMORY BUFFER POOL HIT RATE AS IT INCREASES THE MEMORY
USES BECAUSE MORE AMOUNT OF UNUSED DATA IS FETCHED INTO THE MEMORY. DURING
MEMORY HEAVY OPERATIONS IT WILL USE MORE SWAP-IN AND SWAP-OUT OPERATIONS FOR
THE MEMORY WHICH AGAIN DECREASES THE OPERATION.
• FCC AT THE OTHER HAND DOES ONLY REQUIRED READS AND THUS FETCHES FEWER
AMOUNTS OF DATA IN MEMORY AND INCREASES THE MEMORY BUFFER POOL HIT RATE.
MOREOVER IN SQL SERVER THE FCC IS JUST READ-ONLY HENCE IT MITIGATES THE PROBLEMS
OF THE FCC. MORE TECHNIQUES TO UPDATE THE DATA ARE DISCUSSED IN THE SECTION
BELOW.
Segments & Dictionaries
The columnar indexes are physically stored in the form of Segments. Typically data per column is broken as one million rows per segment (a.k.a. row groups) for each column. The segments are stored as LOB and can contain multiple pages. The index build process runs in parallel and creates as many full segments as possible but some of the segments can have comparatively small size. These segments store highly compressed values
because of the same data types in a segment. Even the large repeated data the compression is even better as a unique small symbol is stored for the duplicate value which saves the size with large degrees. Segments also have header records containing max_data_id, min_data_id etc. These header information is used to omit he compete partition commonly known as segment estimation. The anti-patterns part details even
more about segments. A value reference links to an entry in one of up to two hash Dictionaries. Dictionaries are kept in memory and the data value ids from the segment are referred from these dictionaries but this process is held over as long as possible for better performance reasons. Simply for a table with one partition every column added to the column store index will be added as a row in the segment.
Batch Mode Processing & Row Processing
Query processing can be done either in Row mode or in Batch Mode. While taking an example of join physical join operation takes 2 sets as input parameters and produces the output set based on the join conditions. In the row processing each these sets are processed in row-by-row mode e.g. nested loop join etc. and large amount of CPU is used. Most of the times while operating on large amount of data also spill over
the disk (mostly in hash join); which can be checked by tempdb uses; also increases the memory uses for processing.Vector processing was one of the biggest revolutions which brought the fundamentals of batch processing. These physical operators for query processing takes batch of rows in form of an array (of same type) and process the data. Here batch typically consists of 1000 rows of data
and each column within batch is stored as vector in memory which is known as vector-based query processing. It uses the latest algorithms to utilize the multicore CPUs and the latest hardware. Batch processing works on the compressed data when possible and thus reduces the CPU overhead on join operations; filter etc. (only some of the operators)
www.aditi.com
Demonstration Example
For the demonstration purpose Contoso Retail DW is being used, made available from Microsoft.
Creation of Columnar Index – Code Block 1
CREATE [ NONCLUSTERED ] COLUMNSTORE INDEX index_name ON <tablename> ( column [ ,...n ] )[ WITH ( <column_index_option> [ ,...n ] ) ][ ON { { partition_scheme_name ( column_name ) } | filegroup_name | "default" }]
Below steps can be followed to create the column store index from index creation wizard.
Creation of Columnar IndexColumn Store Indexes Vs. Conventional Indexesquery.
Ultimately they both are same. The basic T-Sql index is as below and details can be captured Column Store
Indexes Vs. Conventional Indexesfrom here.
• EXPAND TILL THE TREE STRUCTURE FOR THE TABLE WHERE COLUMN STORE INDEX IS NEEDED
AND THEN RIGHT CLICK ON THE INDEXES ICON.
• SELECT NEW INDEX AND THEN NONCLUSTERED COLUMNSTORE INDEX AS SHOWN IN DIAGRAM-
6. THIS OPTION WILL BE DISABLED IF COLUMN STORE INDEX ALREADY EXISTS.
www.aditi.com
• CLICK ADD IN THE WIZARD AND IT WILL GIVE YOU A LIST OF COLUMNS WITH CHECK BOXES.
• YOU CAN EITHER CHOOSE COLUMNS INDIVIDUALLY OR CLICK THE BOX NEXT TO NAME AT THE
TOP, WHICH WILL PUT CHECKS NEXT TO ALL THE COLUMNS. CLICK OK. REFER TO THE
DIAGRAM-7.
Graph-9
Graph-8
• CLICK OK. THIS WILL TAKE TIME TO BUILD THE INDEXES BASED ON THE SIZE OF THE TABLE AND
THE COLUMN CHOSEN FOR THE INDEX CREATION.
www.aditi.com
Performance Observations
The performance check was done one the table dbo.FactOnlineSales’ from ContosoRetailDW
having 12.6 million records. More facts and limitations are detailed with example in the
Anti-Patterns section below.
• THE TABLES ‘DBO.FACTONLINESALES’ HAVE MULTIPLE INDEXES AND TO CHECK THE PERFORMANCE
IMPACT OF TABLE ON “B-TREE VS. HEAP” WE CREATED A COPY OF THE SAME TABLE WITHOUT ANY
INDEXES. CORRESPONDING SQL CODE REFERS TO ‘PERFORMANCE OBSERVATION – CODE BLOCK 1’
IN SQL FILE.
• ANALYZE THE QUERY TO GET THE SALES BY PRODUCT WITH THE CONVENTIONAL INDEXES. THE
QUERY USING INDEXES TAKES 5 SEC WHEREAS THE QUERY WITHOUT INDEXES IS A DISASTER
TAKING AROUND 1 MIN AND DOING LOT OF LOGICAL READS. CORRESPONDING SQL CODE REFERS
TO ‘PERFORMANCE OBSERVATION – CODE BLOCK 2’ IN SQL FILE.
• CREATED THE COLUMN STORE INDEX. INDEX CREATION TIME IS DEPENDENT ON THE NUMBER OF
COLUMNS USED IN THE INDEX. ALSO WE SAW THAT IF INDEX IS CREATED BASED ON TABLE HAVING
CLUSTERED INDEX IT DOES VERY LESS READS. MOREOVER IT MAY GET INTERESTING WHEN WE ARE
HAVING 1 BILLION RECORDS IN THE TABLE. WE LEFT THIS ANALYSIS TO THE READERS FOR NOW.
CORRESPONDING SQL CODE REFERS TO ‘PERFORMANCE OBSERVATION – CODE BLOCK 3’ IN SQL
FILE.
• RE-RUN TO GET SALES BY PRODUCT ON WARM CACHE AND CHECK THE EXECUTION TIME AS WELL
AS THE STATISTICS DATA. QUERY EXECUTION WAS ON FIRE WITH WARM CACHE RESULTING LESS
THAN A SECOND OF RESPONSE TIME. THE CORE REASON IS LESS NUMBER OF PHYSICAL READS.
CORRESPONDING SQL CODE REFERS TO ‘PERFORMANCE OBSERVATION – CODE BLOCK 4’ IN SQL
FILE.
• COMPARE THE SALES BY PRODUCTS USING CONVENTIONAL INDEXES AND COLUMN STORE INDEX.
THE COLUMN STORE INDEX TAKES THE 10% OF THE OVERALL EXECUTION PLAN (DIAGRAM-8) AND
ALMOST 1/6 OF THE EXECUTION TIME. IN THE EXAMPLE ABOVE WE HAVE SEEN THAT THE CSI
QUERIES ARE MORE EFFICIENT WITH INCREASING DATA. CORRESPONDING SQL CODE REFERS TO
‘PERFORMANCE OBSERVATION – CODE BLOCK 5’ IN SQL FILE.
Graph-10
www.aditi.com
Design ConsiderationsCandidates for Column Store Index
DW scenarios most commonly fall in the pattern of having read-only data where data is appended periodically commonly using sliding window pattern. They seldom have updates. Data is retained for longer time of at least 8 to 10 years resulting huge volume of data in gigabytes, terabytes or even petabytes for some scenarios. The DW data mostly is divided either in star or snowflake pattern where fact table contains millions and billions or records ready to be aggregate in different fashion. All these schemas are queried typically using star join queries for grouping aggregations.Column store indexes are designed to accelerate the
queries satisfying above said criteria. This makes CSI an absolutely perfect fit for the DW scenarios. So the rule of thumb says large fact tables are the candidates for CSI. Security of the data is not a big concern because CSI also supports Transparent Data Encryption (TDE).Another question is what all columns need to be added to CSI? The answer seems considerably easy that all the columns can be included as long as they follow the prerequisites quoted in the Anti-Patterns section. This decision can be true when we talk about the small or medium scale DW because audit columns or some of the text columns in the fact tables
do not take considerable large space. Although the algorithm is designed to compress in large scale still for the best practice we should only include the all the dimension keys and measures from the table.The fact-less fact tables and multivalued dimensions are not always perfect fit because they will not gain the benefit of batch processing but the advantage of compression and parallel read and segment estimation will definitely be there. Below is the example of choosing candidate tables for CSI. This selection mostly is based upon number of rows and mostly they will be fact tables only.
Candidates for Column Store Index - Code Block 1 Diagram-9 --Choose candidate tables for CSI SELECT O.name TableName ,SUM(P.rows) CountOfRowsFROM sys.partitions PJOIN sys.objects O ON P.object_id = O.object_id WHERE O.type = 'U' --user tablesGROUP BY O.nameORDER BY 2 DESC
Below is the example for choosing candidate columns for the fact table using FactOnlineSales. The mark is used to show the selection for the columns for the dimensions. Along with it all the measures will also be
included while creating CSI. We’ll ignore the audit and degenerated dimension columns here e.g. SalesOrderNumber ,SalesOrderLineNumber ,ETLLoadID ,LoadDate ,UpdateDate. OnlineSalesKey have primary
key defined in it so will automatically be added to the CSI if not mentioned in column list.Corresponding SQL code refers to ‘Candidates for Column Store Index – Code Block 2’ in SQL file.
Graph-11
www.aditi.com
SQL code ‘Candidates for Column Store Index – Code Block 3’ in corresponding SQL file contains the example of the star join
query where results are accelerated within a second. Both the star and snowflake schema query are benefited by CSI.
Snowflake may have issues if any of the primary or and secondary snow flake dimension is too large to support batch mode.
Anti-Patterns
Design considerations are always with-in the defined boundaries. Anti-patterns /
limitations always provide foundations to decide on boundaries making them
the first soprano of design decision.
• Only one CSI can be created on a table. It returns the below error.
• Msg 35339, Level 16, State 1, Line 1• Multiple nonclustered columnstore indexes are not supported.• Key column concept does not relevant in CSI because data is stored in columnar fashion
hence each column will be stored in its own way. Having a clustered key will make difference only while creating the column store index in terms of reads and the order but there is no impact in query performance.
• The base table on which index is created is read-only i.e. can’t be updated or altered. Managing updates is quoted below.
• Interestingly the order of the columns in the create index statement do not have impact either in creating index or in query performance.
• Only limited data types are allowed for CSI i.e.
• CSI can have at most 1024 columns and don’t support
Msg 35339, Level 16, State 1, Line 1Multiple nonclustered columnstore indexes are not supported.
Int, bigint, smallint, tinyint, money, smallmoney, bit, float, real, char(n), varchar(n), nchar(n), nvarchar(n), date, datetime, datetime2, smalldatetime, time, datetimeoffset with precision <=2, decimal/numeric with precision <= 18
Graph-12
www.aditi.com
- Sparse & Computed Columns - Indexed Views or Views - Filtered & Clustered Index - With INCLUDE, ASC, DESC, FORCESEEK keywords - Page and row compression, and vardecimal storage format - Replication, Change tracking, Change data capture & Filestream• CSI can simply be ignored using ‘IGNORE_NONCLUSTERED_COLUMNSTORE_INDEX’. This option helps
user not to know the other index names. Even more helpful if other index names are left to the automatic naming by SQL server when it is difficult to know the name while writing queries.
Anti-Patterns - Code Block 1SELECT P.BrandName Product ,SUM(SalesAmount) SalesFROM dbo.FactOnlineSales SJOIN dbo.DimProduct P ON S.ProductKey = P.ProductKeyGROUP BY P.BrandNameOPTION (IGNORE_NONCLUSTERED_COLUMNSTORE_INDEX)
• Only below operators support batch mode processing therefore use of CSI. - Filter - Project - Scan - Local hash (partial) aggregation - Hash inner join - (Batch) hash table build• Outer joins, MAXDOP 1, NOT IN, UNION ALL are not supported for batch mode execution. Rather we
can have a tweak for the existing query for the same. Some examples are here below.• Although filters in CSI are pushed down to the segments to get benefit of segment estimation but
string filters do not have max or min values hence they do not utilize these filters. So string filters or joins should be avoided on CSI.
• It is observed during above investigation that after the partition switching the query compilation for the first time takes lot of time. Similar behavior was not found while insertion or deletion of data from the table. It may be because of the estimation changes due to partition switching mainly in large data scenario. It is recommended to warm the cache after the partition switching.
Managing Column Store IndexesMemory Considerations
Column Store Indexes (CSI) is the technology which is created considering modern hardware with multiple CPUs and high memory operating on large amount of data specially terabytes. For CSI memory is used during the creation and execution time. We’ll discuss them separately.Creating CSI is a parallel operation. It is dependent on
the available CPUs and the MaxDOP setting restrictions. For large data creation of CSI takes comparatively more time than the B-Tree indexes. Before the creation there is memory estimate done initially for the query execution and that memory grant is provided. There may be cases where initial memory request is not granted and
error code 8657 or 8658 comes. It can be resolved by granting enough memory to the server and the corresponding workgroup. There can be requests for more memory at later point of execution and if enough memory is not there the insufficient memory error can flash i.e. error code 701 or 802.
www.aditi.com
Latter error codes come during the execution at run time whereas the former come at the starting of the query execution. Solution to the problem is to change memory grant for the workgroup or increase memory in the server. The 8657 or 8658 sometimes can occur because of the SQL server configuration of ‘min server memory’ and ‘max server memory’. Suppose the minimum memory needed for the CSI is 3GB and SQL Server have not taken on 1GB memory due to min server memory configuration then it can happen. The resolution can be either run a COUNT(*) query on any of the large tables before the index creation or make the min and max server memory values to same number. This will help SQL server to take the required memory at the
starting time. The resolution for other 2 errors is at 701 and 802. On concluding remarks the CSI can’t be created if enough memory is not there in the system. One of the easiest solutions for such memory considerations is vertical partition of the existing table i.e. breaking the existing table to two or more tables. CSI uses the batch mode processing for the execution. Typically a batch consists of 1000 rows stored in the vector. This type of processing is optimized to use the modern hardware. This provides better parallelism. Batch operators can work on compressed data resulting in high degree of processing in small memory. A considerable amount of memory is needed to execute the batch mode query processing. If the memory is not present the
optimizer changes the query plan to use the row mode. We can check the batch mode processing uses by query plan as below. Batch mode processing always uses the memory so whenever there disk spill due to large data the row by row processing replaces the batches; mostly seen during hash join for large tables. Another reason for row by row processing is incorrect statistics update which in turn spills the data into the disks resulting row by row operation. To check this operation the extended event batch_hash_table_build_bailout’ can be configured. The warning ‘Operator used tempdb to spill data during execution’ also flashes for this kind of behavior.
Add & Modify Data in Column Store IndexTable with CSI is read only i.e. we can’t perform operations like INSERT, UPDATE, DELETE or MERGE. These operations fail with the error message
e.g.
Msg 35330, Level 15, State 1, Line 1UPDATE statement failed because data cannot be updated in a table with a columnstore index. Consider disabling the columnstore index before issuing the UPDATE statement, then rebuilding the columnstore index after UPDATE is complete.
Graph-13
www.aditi.com
Considering this we have the below options or workarounds for the operation.
• Have staging/work tables without CSI (most of the cases these are drop and recreate tables). Create CSI and switch it to the empty partition of the table. We have to make sure that we have the empty partition because if there is data in the partition and CSI is created into the table we can’t split it. Below is the example code segment for the same. Corresponding SQL code refers to ‘Add & Modify Data – Code Block 2’ in SQL file.
• Switch a partition from table to the empty staging table. Drop CSI from staging table and perform updates, inserts etc. and build the CSI and switch the staging table to the empty (empty by previous switch) partition. Corresponding SQL code refers to ‘Add & Modify Data – Code Block 3’ in SQL file.
• We can choose to create different underline tables to represent a fact table and access all of them using UNION ALL views. Just disable the index in the most recent table which will have the updates and rebuild/recreate the CSI. We can always get the data from those UNION ALL views.
• Put the data into the staging table, create the CSI in staging table and just drop the existing table and rename the staging table to original (better to do both of the operations in a transaction, note that both of them will be metadata operations only). This will have the more processing time but will ensure the high availability. This option can be chosen only when there are relatively small or medium scales of data in the table.
Add & Modify Data - Code Block 1ALTER INDEX csiFactOnlineSales ON dbo.FactOnlineSales DISABLEGOUPDATE dbo.FactOnlineSalesSET SalesAmount = SalesAmount * 2GOALTER INDEX csiFactOnlineSales ON dbo.FactOnlineSales REBUILD
Size of Column Store IndexSize of the CSI is based on the size of the segment and dictionaries. Most of the space is used by the segments. We can get the same in more simplified manner. Here are the simple and the actual size estimation query.
Statistics are another valuable consideration. We have statistics for the base table having CSI but not for the CSI in particular. The statistics object is created for the CSI but SHOW_STATISTICS shows null for the CSI and show values for the clustered index.
The statistics object for CSI is used for the database cloning (DB clone is copy of the statistics-only database investigating query plan issues). Corresponding SQL code refers to ‘Size of Column Store Index – Code Block 1’ in SQL file.
www.aditi.com
Column Store Indexes Vs. Conventional IndexesColumn Store Index vs. Clustered Indexes
CSI is different than all other conventional indexes. Both of them are the utilities for different type of scenarios. Till now we have seen that CSI are a lot faster than the conventional indexes. Here below is the example where CSI is taking almost 99% in the relative query plan. Here
we are using the highly selective query i.e. only few records are being queried using both of the indexes. Please take a note that we are using only those columns which are being used in the CSI creation. The comparisons among the indexes is always based on the nature of uses
on the data i.e. queries. SQL Server automatically figures out the highly utilized query. Moreover plan guides can also be pinned for abnormal behavior of the queries. Here for the apple to apple comparison we are using only the columns used to create CSI.
Column store index Vs. Covering Indexes Vs. One index each columnCovering index is the highly used terminology to achieve the high performing queries. Creation of covering index is always a cautious decision. It is very difficult to put indexes which cover all the queries, particularly in the data warehousing scenarios where users are open to use any kind of queries. Covering index can be achieved either by adding the columns into the index i.e. composite index or by pinning them to the B-Tree using
INCLUDE keyword. A very detailed description can be referred from here.CSI or the covering index, again the discussion depends on the amount of data, the query and the memory. On the same nodes CSI uses compression as well as the batch mode processing hence faster scans. If we have the entire star schema for our DW the CSI is best to use for aggregative queries. It also reduces the index design and maintenance time and one index shows all of the magic.
Selecting one more column can make the covering index ineffective which is not the case with normal index. Creating each index each column will not be useful on selecting multiple columns. Moreover the size of all the covering or other indexes captures relatively larger footprint on the disk, which is multiple copies of the same data resulting more maintenance and sometimes adding to downtime to the application.
Column Store Index vs. Clustered Indexes – Code Block 1SELECT SalesAmount ,ProductKey FROM dbo.FactOnlineSales S WITH (INDEX(PK_FactOnlineSales_SalesKey))WHERE OnlineSalesKey IN (32188091,23560484,31560484,27560484) SELECT SalesAmount ,ProductKey FROM dbo.FactOnlineSales S WITH (INDEX(csiFactOnlineSales))WHERE OnlineSalesKey IN (32188091,23560484,31560484,27560484)
Graph-14
www.aditi.com
On the other hand here is another example which shows that CSI is not benefiting more on query execution time because of the large hash joins and batch execution turning back to row by row
execution. Here we’ll just create example table joining with FactOnlineSales. Both of the table will have the same cardinality. We can easily see a warning message and warning icon in the actual
query plan. Corresponding SQL code refers to ‘Column store index Vs. Covering Indexes – Code Block 1’ in SQL file.
TempDB is core of all the temp operations for which memory is not granted. SQL server uses the tempdb extensively and if users have read only permissions on any of the databases that ensures the read-write permissions on the tempdb as well. Point of
analysis is tempdb uses during creating and query on CSI. For the surprise the tempdb was not used during the creation as well as querying time. The tempdb will be used when the execution is done using row by row operation instead of
batch ensuring data is spilled into the disk i.e. tempdb is used and above is the example showing this behavior. Corresponding SQL code refers to ‘Analyzing TempDB Uses – Code Block 1’ in SQL file.
Performance Tuning ConsiderationsAnalyzing TempDB Uses
Graph-15
Graph-16
www.aditi.com
Maximizing Segment EstimationThe data of CSI is divided in segments and this information is stored in the ‘column_store_segments’ system table. The columns for the relevance to understand segment estimation are in below query.
Here segment stores the min and max value for the segment and if the filter value does not belongs to the
segment the scan for that segment is ignored i.e. called segment estimation. E.g. if we are writing a query for the
below which says ‘OnlineSalesKey > 30000000’ the second segment will be ignored.
Here in the example we are seeing that the min and max values are skewed. This is not ideal for the segment estimation because only one
segment is eliminated. Here we need to find how to arrange the data so that we have maximum number of partitions and the values are
aligned properly to the segments. We can use the below techniques.
Maximizing Segment Estimation - Code Block 2SELECT G.ContinentName ,SUM(S.SalesAmount) TotalSales FROM dbo.FactOnlineSales SJOIN dbo.DimCustomer C ON C.CustomerKey = S.CustomerKeyJOIN dbo.DimGeography G ON G.GeographyKey = C.GeographyKeyWHERE S.DateKey BETWEEN '2012-01-01' AND '2012-12-30'GROUP BY G.ContinentName
Maximizing Segment Estimation - Code Block 1SELECT S.column_id ,S.segment_id ,S.min_data_id ,S.max_data_id FROM sys.column_store_segments S
• MOST OF THE CASES DATA IS ACCESSED BY DATE SO WE CAN PRE-SORT THE DATA BASED ON THE
DATE AND THUS WE ARE CREATING PARTITIONS BASED ON THE DATE. WE ACHIEVE THIS BY
CREATING CLUSTERED INDEX BEFORE CREATING A CSI AND THUS THE DATA WILL BE ALIGNED TO
THE DATE AND SEGMENT ESTIMATE IS MAXIMIZED.
• ANOTHER INNOVATIVE METHOD CAN BE CREATING PARTITIONS BASED ON THE DIFFERENT
COMBINATIONS OF DIMENSIONS. LET US TAKE AN EXAMPLE OF THE BELOW QUERY WHERE WE
HAVE TO GET THE DATE FILTER AND THE SUM OF ALL SALES ACROSS THE GEOGRAPHY. WE NEED TO
CREATE ONE MORE CLUSTERED KEY BASED ON THE YEAR AND GEOGRAPHY AS SHOWN IN TABLE.
THIS WAY WE’LL GET 9 PARTITIONS FOR 3 YEARS AND 3 GEOGRAPHY VALUES. WE NEED TO CREATE
A CLUSTERED INDEX ON THE NEWLY BUILD CLUSTERED KEY AND CREATE CSI ON IT AND RUN THE
SAME QUERY.
Graph-16
www.aditi.com
On running the above query again we can found that the segment estimation will scan will skip the crossed partitions and thus the segment estimation is maximized. It is nice to use this approach but it is very hard to manage these kinds of partitions and it may end up coming out to be another tool. Moreover adding other multiple dimensions will add similar complexity to the
partitions. We also should have enough data for each partition so that the segments are utilized. If we’ll have less than 1 million records we may end up doing crash landing and queries may not help as expected.
Ensuring Batch Mode ExecutionBatch mode vector based execution helps the query a
lot. MAXDOP configuration helps to check this behavior by ensuring that none of the queries are using MAXDOP 1 option. Below example shows the difference in the execution plan. The below query plan shows that there is no use of parallel and batch operations. Moreover the cost for the CSI scan is also more for MAXDOP 1.
Batch mode processing is not supported for outer joins for this release of SQL Server. To get the benefit of the batch processing we need to change the queries a bit. One of the typical example of the changing the query is as below where we first are getting inner join values and joining them back to the dimension table for outer join
records. The query plan shows all different results where batch mode and row mode is used along with the parallelism. It also shows that the alternate query just takes 12% of the relative cost.These examples shows that we need to redesign our conventional queries to take advantage of the batch mode. The bottom line is we have to
mark time and have close eye for each query being written on CSI. Query plans should be monitored closely for further changes not only in development but also in production environments. Corresponding SQL code refers to ‘Ensuring Batch Mode Execution – Code Block 1’ in SQL file.
Graph-17 Graph-18
Graph-19
www.aditi.com
Graph-20
Graph-21
www.aditi.com
About AditiAditi helps product companies, web businesses and enterprises leverage the power of cloud, e-social and mobile, to drive competitive advantage. We are one of the top 3 Platform-as-a-Service solution providers globally and one of the top 5 Microsoft technology partners in US.
We are passionate about emerging technologies and are focused on custom development. We provide innovation solutions in 4 domains:
Digital Marketing solutions that enable online businesses increase customer acquisitionCloud Solutions that help companies build for traffic and computation surgeEnterprise Social that enables enterprises enhance collaboration and productivityProduct Engineering services that help ISVs accelerate time-to-market
www.aditi.com
https://www.facebook.com/AditiTechnologies
http://www.linkedin.com/company/aditi-technologies
http://adititechnologiesblog.blogspot.in/
https://twitter.com/WeAreAditi
Using PartitioningPartitions are another key performance factor which still works with columnar indexes having said that every non-partitioned table always has one physical partition. We can create partitions on the table and then create CSI. The CSI must be partition aligned with the table partition. CSI uses the same partition scheme used by the base table and equal number of partitions will be created. We can also switch in or switch out the partition and that will partition the corresponding CSI. Segments are created on the partitions. So if the partitions are skewed we can see more number of segments. Modifying CSI using partitioning is detailed
here.Every partition is compressed separately and has its own dictionaries. These dictionaries are shared across the segments within same partition. We can easily find which segments and dictionaries belong to any specific partition. Thus partition switching still is a metadata operation. We already have partitions created on the table while exploring CSI management above and will be using same to explore more. Below example shows that we have 2 dictionaries for each partition irrespective of segments except partition 5 which is without dictionaries. Exploring this behavior is left to the reader.
ConclusionsColumnar indexes are the breakthrough innovation with the capability to expend the envelope to improve overall performance of the ETL and DW workload. Current version of 2012 has a bit of limitation with expectation of improvement in future version especially with respect to addition and modification of data. It’s recommended to use the best practices, understanding case studies and fitting the defined business problem to the columnar solution pattern.
Using Partitioning - Code Block 1 Diagram-21
/*Exploring Partitions*/SELECT * FROM sys.partition_schemesSELECT * FROM sys.partition_functionsSELECT * FROM sys.partition_range_valuesSELECT * FROM sys.partition_parameters SELECT P.partition_number ,COUNT(*) Segment#FROM sys.column_store_segments SJOIN sys.partitions P ON P.hobt_id = S.hobt_idWHERE P.object_id = OBJECT_ID('[dbo].[FactOnlineSales]')GROUP BY P.partition_number SELECT P.partition_number ,COUNT(*) Dictionary# FROM sys.column_store_dictionaries DJOIN sys.partitions P ON P.hobt_id = D.hobt_idWHERE P.object_id = OBJECT_ID('[dbo].[FactOnlineSales]')GROUP BY P.partition_number
