CacheMind:Fast Performance RecoveryUsing a Virtual Machine Monitor

Kenichi KouraiKyushu Institute of Technology, Japan

OS RecoveryOS reboot is a final but powerful recovery

techniqueFor recovery from OS crashes

Against MandelbugsA rebooted OS rarely crashes again

For software rejuvenationAgainst aging-related bugsA rebooted OS restores

its normal staterecovered

OSreboot

reboot

memoryleak

crash

Performance Degradation (1/2)OS reboot degrades the performance of

file accessesThe file cache on memory is lost

Disk access increases due to frequent cache misses

It takes long time to fill the file cacheReading file blocks from a disk is slowMost of free memory is used for the file cache

file cache

reboot slow disk

Performance Degradation (2/2)Disk access also degrades the

performance of the other virtual machines (VMs)VMs share a physical disk

Frequent disk access occupies the bandwidthPrefetching makes the situation worse

Burst of disk accessVM VM

disk

rebootedVM

OS

Performance Recovery is NeededOS recovery does not complete until the

performance is also recoveredTraditional OS reboot restores only the

functionalitiesFast reboot techniques have been proposed

Warm-cache RebootA new OS recovery mechanism with fast

performance recoveryIt preserves the file cache during OS reboot

An OS can reuse it after the rebootIt guarantees the consistency of the file

cacheUsing the virtual machine monitor (VMM)

filecache

reboot

filecache

VMM

discardVM

corruptedcache

Reusing the File CacheCollaboration between an OS and the VMMThe VMM re-allocates the same physical

memory to a rebooted VMA rebooted OS reserves the memory pages

used for the file cacheObtaining meta data from the VMM

filecache

VMM

reboot

filecache

re-allocate

reserve

deallocate

VM

Cache ConsistencyOur definitionConsistent if the contents of the file cache

are the same as those of disksConsistent when a file block is read from a diskInconsistent when the file cache is modifiedConsistent when it is written back to a disk

diskfile cache

modify

read

write backVM

Maintaining Cache ReusabilityThe warm-cache reboot allows an OS to

reuse only consistent file cacheThe VMM is suitable for maintaining the

reusabilityIt is isolated from an OSIt can mediate all disk accessesIt can track all modification to cache pages

VMM

VM

disk

modify cachepages

file cache

Reusability Management (1/3)The VMM makes a cache page reusable

after it reads data from a diskIt protects the page before the read

To detect page corruption by an OS during the read

The VMM can still write data to the page

VMM

read

VM

read request

protect read reusable

readrequest

possiblecorruption

disk

Reusability Management (2/3)The VMM makes a cache page non-

reusable before an OS modifies its contentsIt unprotects the page at the same time

To enable the OS to modify the page

VMM

VM

modify request

unprotect

non-reusable &unprotect

modifyrequest write

possiblecorruption

Reusability Management (3/3)The VMM makes a cache page reusable

again after it writes data in the page to a diskIt protects the page before the write

To detect page corruption during the write

VMM

write

VM

write request

protect write reusable

writerequest

possiblecorruption

disk

ImplementationCacheMindBased on Xen/LinuxPer-VM data

Hash table from file blocksto cache pagesDomain U adds/removes

Bitmap for reusingcache pagesDomain 0/VMM change

blkback blkfront

domain 0 domain U

VMM

disk

Per-VMdata

ExperimentsPurposesTo show that the warm-cache reboot

achieves fast performance recoveryFile access, web server

To confirm that it does not reuse inconsistent file cachefault injection

ServerCPU: 2 dual-core OpteronMemory: 12 GBDisk: Ultra 320 SCSINIC: Gigabit Ethernet

ClientCPU: 2 Core 2 QuadMemory: 4 GBNIC: Gigabit Ethernet

Throughput of File ReadsWe measured the read throughput of a 1-

GB fileAll file blocks were on the file cache

1st 2nd 3rd 4th 5th 6th0

200400600800

100012001400

normal rebootwarm-cache reboot

thro

ughp

ut (M

B/s)

before reboot after reboot

Our reboot achievedbetter performance

16% degradationat maximum

Throughput of a Web ServerWe measured the changes of the

throughput during OS reboot

60% degradationfor 90 seconds

5% degradationfor 60 seconds

Fault Injection (1/2)We measured inconsistent cache reusesWe injected various faults into the OS kernelFirst, we disabled the consistency

mechanism

DST INIT BRPA

NICFR

EECOPY

STAC

K0

1020304050607080

no crashprocess crashkernel crash

inco

nsis

tent

reu

se

(%)

The file cache isoften corrupted

Fault Injection (2/2)Next, we enabled the consistency

mechanismMost of reboots did not reuse inconsistent

cacheReused file cache was inconsistent only for

DSTExt3 failed to write back

Faults were injectedinto ext3

The file cache was notcorruptedReusing it is correct DST

05

1015202530354045

disabledenabled

inco

nsis

tent

reu

se

(%)

Related WorkRio File Cache [Chen et al.’96]Reusing dirty file cache after OS crashRelying on an OS

RootHammer [Kourai et al.’07]Preserving VMs during VMM reboot

Hybrid Hard Drive [Samsung&Microsoft],Turbo Memory [Intel]Including large non-volatile disk cache

ConclusionWe proposed the warm-cache rebootIt achieves fast performance recovery by

reusing the file cache16% degradation at maximum

The VMM maintains consistency of the file cacheConsistent, or not-corrupted at least

Future workOverhead measurement of the consistency

mechanism