Columnar Flash Cache

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Columnar Flash CacheAdrian Ng and Selcuk Aya


Agenda• Problem• High-level Idea• Columnar Cache Characteristics• Deep Dive on the FC layer• Future Enhancements and others• Questions


Problem• Problem

– Smart scan on wide EHCC table can get IO-bound

• Why specific to wide table?– The percentage of data for a given column in a CU

for a wide table is small compared to narrow table.– More CUs are fetched from disks/flash to get data

for the entire column– Queries reading only a few columns of a wide

EHCC table exhibit high IO bandwidth utilization due to a lot of irrelevant columns being read from storage

Oracle Confidential – Internal/Restricted/Highly Restricted 3

Col 1 Col 2

Col 3

Col 98 Col 99 Col 100

….

Col 1 Col 2

Col 3


….

Col 1 Col 2

Col 3


….

.

.

.

1st CU in a 1MB region

2nd CU in a 1MB region

Nth CU in a 1MB region

1MB worth of EHCC blocks




High-level Idea (1 of 3)• Have a columnar representation of data on flash, as cache• On smart scan, read columns that are relevant to queries• With this, we can avoid doing unnecessary IOs and hence avoid the wide-

table problem. • For this, we implemented columnar cache in 12.1.2.1. • This project was originally implemented by Selcuk Aya, Roger Macnicol,

Viral Shah, Minji Kim and myself.



High-level Idea (2 of 3)

Oracle Confidential

Col 1 Col 2

Col 3


….

1MB EHCC buffer

Col 1 Col 2

Col 3


….

Col 1 Col 2

Col 3


….

Metadata Header

Col 1

Col 2

Col 1

Col 2

Col 3

Col 100

Col 99

Columnar Cache Format on flash

1st 64KB Cache line for the 1MB

.

.

.




. . .

2nd 64KB Cache line for the 1MB

3rd 64KB Cache line for the 1MB

16th 64KB Cache line for the 1MB

. . .

Rewrite: Same 1MB data, but laid out differently in 64KB


High-level Idea (3 of 3)

Oracle Confidential

Col 1 Col 2

Col 3


….

1MB EHCC buffer

Col 1 Col 2

Col 3


….

Col 1 Col 2

Col 3


….

Metadata Header

Col 1

Col 2

Col 1

Col 2

Col 3

Col 100

Col 99

Columnar Cache Format on flash

1st 64KB Cache line for the 1MB

.

.

.




. . .

2nd 64KB Cache line for the 1MB

3rd 64KB Cache line for the 1MB

16th 64KB Cache line for the 1MB

. . .

Select max(col99) from table

< --- IO Pruned!

< --- IO Pruned!




Columnar Cache Characteristics (1 of 2)• Useful only for analytics workload• Same 1MB region can be cached both in “block” format for OLTP and

“columnar” format for DW. • Populated automatically during EHCC-table smart scan, subject to the same

FC caching (e.g. auto caching) policies• Invalidated upon writes• Entire 1MB columnar cache is treated as one unit for hotness tracking and

eviction• Cache is not persistent – gone upon restart



Columnar Cache Characteristics (2 of 2)• Visible in CELLCLI just like “Block Cache”

CellCLI> list flashcachecontent where objectnumber=92598 detail cachedKeepSize: 0 cachedSize: 104857600 cachedWriteSize: 0 columnarCacheSize: 104857600 columnarKeepSize: 0 dbID: 1896360620 dbUniqueName: MAIN2 hitCount: 0 missCount: 0 objectNumber: 92598 tableSpaceNumber: 1

•





High Level Architecture

• Columnar caches underlying disk region in columnar format

• Columnar Cache is a separate cache living on top of Flash Cache– Uses the same flash device area as Flash Cache– Shares significant amount of code with Flash Cache– No hard space partitioning between Flash Cache and

Columnar Cache– Columnar Cache and Flash Cache compete for space and

Lru decides what is cached

Columnar Cache Layering


Columnar Cache

Flash Cache

Disk

Colum

nar C

ache M

iss

Flash C

ache IO

Colum

nar C

ache IO

Client

Client

IO



• Same disk region can be cached both in Columnar Cache and Flash Cache

• Can be improved to store any transformation of the underlying disk region

• Writethrough– Writes invalidate columnar cache

• Not persistent– Might be improved to be persistent

Columnar Cache Layering


Columnar Cache

Flash Cache

Disk

Colum

nar C

ache M

iss

Flash C

ache IO

Colum

nar C

ache IO

Client

Client

IO



• If we have a client IO which can handle columnar format, first poke Columnar Cache

• If IO hit, read from Columnar Cache.• If IO miss, submit IO to Flash Cache

– After IO completes try to populate columnar cache

IO Flow


Columnar Cache

Flash Cache

Disk

Colum

nar C

ache M

iss

Flash C

ache IO

Colum

nar C

ache IO

Client

Client

IO


Data Structures

• Reuse Flash Cache Data Structures

• Columnar Cache header and its child Cache Headers keep track of columnar format of underlying 1MB disk region

• Hash Table and Lru are common for Columnar Cache and Flash Cache


FC HashTabl

e

GD_OO, offset a

Child Cache header

...

GD_OO, offset a

Columnar Cache Header andIts Child Headers

Flash CacheHeader


Data Structures

• Columnar Cache Header keeps track of Lru hotness and validness of the columnar format of the region

• In the code, IOs to the columnar format of the region synchronize through Columnar Cache Header

(Continued..)


GD_OO, offset a

Child Cache header

...

GD_O1, offset a

Columnar Cache Header and Its Child Headers

Lru


IO Flow

• A scenario where we have a miss in columnar cache and hit in flashcache– 1) Poke Columnar Cache, conclude we have miss. Get a

Columnar Cache Header and pin this Cache Header for potential population and remember this cache header

– 2) Submit IO to Flash Cache. We have a hit. Pin Flash Cache cache headers and read from them.

– 3) IO at step 2 completes and notifies Client.– 4) Smart Scan layer transforms data to columnar format and

asks Columnar Cache to populate Columnar Cache

Columnar Cache Miss and Population


Columnar Cache

Flash Cache

Disk

Colum

nar C

ache M

iss

Flash C

ache IO

Colum

nar C

ache IO

Client

Client

IO


IO Flow

• A scenario where we have a miss in columnar cache and hit in flashcache– 5) Columnar Cache finds enough child cache headers to

cache columnar format and submits IO for population– 6) IO to populate Columnar Cache completes and Columnar

Cache Header is marked valid.

Columnar Cache Miss and Population(Continued..)


Columnar Cache

Flash Cache

Disk

Colum

nar C

ache M

iss

Flash C

ache IO

Colum

nar C

ache IO

Client

Client

IO


IO Flow

• A scenario where we have a miss in columnar cache and miss in flashcache– 1) Poke Columnar Cache, conclude we have miss. Get a

Columnar Cache Header and pin this Cache Header for potential population and remember this cache header

– 2) Poke Flash Cache. We have a miss. Find and pin all cache headers for potential population of the disk region. Then submit IO to disk

– 3) IO at step 2 completes. Client is notified– 4) Smart Scan layer prepares columnar format and asks

Columnar Cache to populate columnar cache

Columnar Cache Miss, Flash Cache Miss


Columnar Cache

Flash Cache

Disk

Colum

nar C

ache M

iss

Flash C

ache IO

Colum

nar C

ache IO

Client

Client

IO


IO Flow

• A scenario where we have a miss in columnar cache and miss in flashcache– 5) At this point, columnar cache releases all cache headers

pinned for potential Flash Cache population and then finds enough child cache headers for Columnar Cache population. It then submits IO to populate Columnar Cache

– 6) IO to populate Columnar Cache is submitted– 7 When IO at step 6 completes, Columnar Cache Header is

marked valid.

Columnar Cache Miss, Flash Cache Miss(Continued…)


Columnar Cache

Flash Cache

Disk

Colum

nar C

ache M

iss

Flash C

ache IO

Colum

nar C

ache IO

Client

Client

IO


IO Flow

• A scenario where we have a miss in columnar cache and miss in flashcache

• In our example, smart scan layer might have decided region cannot be transformed to columnar format– In this case it would abort Columnar Cache population

which would release Columnar Cache Header – Then it would populate Flash Cache– This scenario is why we pin both Columnar and Flash Cache

cache headers for potential population

Columnar Cache Miss, Flash Cache Miss(Continued…)


Columnar Cache

Flash Cache

Disk

Colum

nar C

ache M

iss

Flash C

ache IO

Colum

nar C

ache IO

Client

Client

IO


IO Flow

• Client submits IO and can deal with columnar IO– Poke Columnar Cache and conclude we have a hit. Pin the

Columnar Cache Header. – Ask client which columns it is interested in– Read the cache headers that contains columns of interest

• Save on the IO and CPU cost of submitting/reaping IOs

Columnar Cache IO Hit


Columnar Cache

Flash Cache

Disk

Colum

nar C

ache M

iss

Flash C

ache IO

Colum

nar C

ache IO

Client

Client

IO


Agenda• Problem• High-level Idea• Columnar Cache Characteristics• Deep Dive on the FC layer• Future Enhancements and others • Questions


Future enhancements • Caching blocks in IMC format instead of EHCC• Caching selective columns• Persist columnar cache across cellsrv restarts



Others• Release information


Product Minimum Supported Version

RDBMS 12.1.0.2.0

OSS 12.1.2.1.0

• To disable columnar cache - _enable_columnar_cache = 0•


Questions?• Q&A


Documents

Columnar Flash Cache