Embed Size (px)
Citation preview
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Amazon Redshift Deep Dive
Ran Tessler, AWS Solutions Architect
Amazon Redshift Architecture
• Leader Node
– SQL endpoint
– Stores metadata
– Coordinates query execution
• Compute Nodes
– Local, columnar storage
– Execute queries in parallel
– Load, backup, restore via
Amazon S3; load from
Amazon DynamoDB or SSH
• Two hardware platforms
– Optimized for data processing
– Dense Storage: HDD; scale from 2TB to 2PB
– Dense Compute: SSD; scale from 160GB to 326TB
10 GigE
(HPC)
Ingestion
Backup
Restore
JDBC/ODBC
• Massive Parallel Processing (MPP)
– Nodes are split into independent slices
– Each slice has a single virtual core, dedicated RAM
and storage
Amazon Redshift Architecture
10 GigE
(HPC)
Ingestion
Backup
Restore
JDBC/ODBC
Compute Node
Slice 1 Slice 2
Virtual Core
7.5 GiB RAM
Local Disk
Virtual Core
7.5 GiB RAM
Local Disk
Amazon Redshift dramatically reduces I/O
• Data compression
• Zone maps
• Direct-attached storage
• Large data block sizes
ID Age State Amount
123 20 CA 500
345 25 WA 250
678 40 FL 125
957 37 WA 375
Amazon Redshift dramatically reduces I/O
• Data compression
• Zone maps
• Direct-attached storage
• Large data block sizes
ID Age State Amount
123 20 CA 500
345 25 WA 250
678 40 FL 125
957 37 WA 375
Amazon Redshift dramatically reduces I/O
• Column storage
• Data compression
• Zone maps
• Direct-attached storage
• Large data block sizes
analyze compression listing;
Table | Column | Encoding
---------+----------------+----------
listing | listid | delta
listing | sellerid | delta32k
listing | eventid | delta32k
listing | dateid | bytedict
listing | numtickets | bytedict
listing | priceperticket | delta32k
listing | totalprice | mostly32
listing | listtime | raw
Amazon Redshift dramatically reduces I/O
• Column storage
• Data compression
• Direct-attached storage
• Large data block sizes
• Track of the minimum and
maximum value for each block
• Skip over blocks that don’t
contain the data needed for a
given query
• Minimize unnecessary I/O
Amazon Redshift dramatically reduces I/O
• Column storage
• Data compression
• Zone maps
• Direct-attached storage
• Large data block sizes
• Use direct-attached storage
to maximize throughput
• Hardware optimized for high
performance data
processing
• Large block sizes to make the
most of each read
• Amazon Redshift manages
durability for you
Data Modeling
Data Distribution
• Data is allocated to slices based on
distribution style
– DISTSTYLE EVEN – Round Robin
– DISTSTYLE KEY – based on the distribution key
hash value
– DISTSTYLE ALL - Replicated to slice 0 on all
nodes
• Query performance considerations
– Uneven distribution harms query
– Data redistribution is expensive
Compute Node 1
Slice 1 Slice 2
Compute Node 2
Slice 3 Slice 4
5M
records
2M
records 1M
records
4M
records
Compute Node 1
Slice 1 Slice 2
Compute Node 2
Slice 3 Slice 4
Compute Node 3
Slice 5 Slice 6
Suboptimal Distribution
ORDERS ITEMS
Default (No Distribution Key, Round Robin Order)
Order 1 Order 2 Order 3Item 2.1 Item 1.1 Item 1.2
Item 2.2Item 3.1
Order 1: Dave Smith, Total $195
Item 1.1: Order 1, Kindle Fire HD 7”, $159
Item 1.2: Order 1, Kindle Fire Case, $36
Compute Node 1
Slice 1 Slice 2
Compute Node 2
Slice 3 Slice 4
Compute Node 3
Slice 5 Slice 6
Optimal Distribution
ORDERS ITEMSOrder 1: Dave Smith, Total $195
Item 1.1: Order 1, Kindle Fire HD 7”, $159
Item 1.2: Order 1, Kindle Fire Case, $36
Order 1 Order 2 Order 3
Item 2.1Item 1.1
Item 1.2 Item 2.2
Item 3.1
Customised (ORDERS.ORDER_ID DISTKEY, ITEMS.ORDER_ID DISTKEY)
Sorting Table Data
• Sort Keys ≠ Index
– Data is initially written by INSERT/COPY order
– VACUUM sorts the rows and reclaims stale
storage
Compound Sort Keys Illustrated
Records in Redshift are stored in blocks.
For this illustration, let’s assume that four records fill a block
Records with a given cust_id are all in one block
However, records with a given prod_id are spread across four blocks
1
1
1
1
2
3
4
1
4
4
4
2
3
4
4
1
3
3
3
2
3
4
3
1
2
2
2
2
3
4
2
1
1 [1,1] [1,2] [1,3] [1,4]
2 [2,1] [2,2] [2,3] [2,4]
3 [3,1] [3,2] [3,3] [3,4]
4 [4,1] [4,2] [4,3] [4,4]
1 2 3 4
prod_id
cust_id
cust_id prod_id other columns blocks
1 [1,1] [1,2] [1,3] [1,4]
2 [2,1] [2,2] [2,3] [2,4]
3 [3,1] [3,2] [3,3] [3,4]
4 [4,1] [4,2] [4,3] [4,4]
1 2 3 4
prod_id
cust_id
Interleaved Sort Keys Illustrated
Records with a given
cust_id are spread across
two blocks
Records with a given
prod_id are also spread
across two blocks
Data is sorted in equal
measures for both keys
1
1
2
2
2
1
2
3
3
4
4
4
3
4
3
1
3
4
4
2
1
2
3
3
1
2
2
4
3
4
1
1
cust_id prod_id other columns blocks
Query Optimization
Query Performance
• Good choice of distribution and sort keys speed query
performance more than any other factor
• Redshift Uses a Cost Based Query Optimizer
– Good statistics are VITAL to ensure good performance
– Table constraints, while not enforced, are used to optimize queries
• Run ANALYZE command to update statistics:ANALYZE lineitem;
Query Analysis
• EXPLAIN command followed by the query:EXPLAIN select avg(datediff(day, listtime, saletime)) as avgwait from sales,
listing where sales.listid = listing.listid;
QUERY PLAN
XN Aggregate (cost=6350.30..6350.31 rows=1 width=16)
-> XN Hash Join DS_DIST_NONE (cost=47.08..6340.89 rows=3766 width=16)
Hash Cond: ("outer".listid = "inner".listid)
-> XN Seq Scan on listing (cost=0.00..1924.97 rows=192497 width=12)
-> XN Hash (cost=37.66..37.66 rows=3766 width=12)
-> XN Seq Scan on sales (cost=0.00..37.66 rows=3766 width=12)
• From the EXPLAIN plan you can tell:
– Query execution steps
– Which operation to be performed in each step
– Which table to be used in each step
– How much data needs to be processed in each step
Query Analysis
• Access the STL_EXPLAIN table for executed queries: select query,nodeid,parentid,substring(plannode from 1 for 30),
substring(info from 1 for 20) from stl_explain
where query=10 order by 1,2;
query | nodeid | parentid | substring | substring
------+--------+----------+---------------------+---------------------
10 | 1 | 0 | XN Aggregate (cost=6350.30... |
10 | 2 | 1 | -> XN Merge Join DS_DIST_NO | Merge Cond: ("outer"
10 | 3 | 2 | -> XN Seq Scan on lis |
10 | 4 | 2 | -> XN Seq Scan on sal |
• SVL_QUERY_SUMMARY and SVL_QUERY_REPORT for finer details
Query Analysis
• Explain plans and performance metrics are also available via
the console:
Query Analysis
• Explain Plan Visualization is now also available
Amazon Redshift
Spend time with your data, not your database….
Thank You
