An SMB3 Engineer’s View of Windows Server 2012 Hyper-V ......Windows 2012: Command Run1 Run2 Run3 Run1 Run2 Run3 Run1 Run2 Run3 Negotiate 1 1 1 1 1 1 1 1 1 SessionSetup 60 60 60

2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.

An SMB3 Engineer’s View of Windows Server 2012 Hyper-V Workloads

Gerald Carter [email protected]

EMC/Isilon


One sunny day in San Jose….

FAST 2013 - “Virtual Machine Workloads: The Case for New NAS Benchmarks” V.Tarasov, Stony Brook University; D.

Hildebrand, IBM Research—Almaden; G. Kuenning, Harvey Mudd College; E. Zadok, Stony Brook University

https://www.usenix.org/conference/fast13/virtual-machine-workloads-case-new-nas-benchmarks

2


Questions…

Do the existing characterizations of NFS data stores for hypervisors apply to Hyper-V and SMB3?

What do the I/O and jump distance patterns look like?

What SMB3 protocol features does Hyper-V use?

3


Addendum to Abstract…

Original scope of abstract was reduced due to time constraints and hardware availability SMBDirect Failover Multi-Channel

Addition of tools for packet capture introspection

4


Test Setup

5


Test Bench Specifications

Hardware Intel Quad CPU Q9650 @ 3GHz, 16Gb RAM,

SATA II

Networking Cisco SG-200 1Gbps switch

Environment Windows Server 2012 Standard Hyper-V and

File Services

6


VM Configuration

7

Operating System RAM CPU Disk (Thin)

Ubuntu 12.04 x64 1 GB 1x vCPU (4x vCPU)

40 GB IDE

Windows XP SP3 x86

512 MB 1x vCPU

40 GB IDE

Windows Server 2012 Standard

2 GB 1x vCPU

60 GB IDE (60 GB SCSI)


Deployment to SMB File Share

8

PS> Import-VM -path $SourceXml -copy -GenerateNewID -VirtualMachinePath $TargetDir -SnapshotFilePath $TargetDir -SmartPagingFilePath $TargetDir -VhdDestinationPath “$TargetDir\Virtual Hard Disks\” PS> Move-VMStorage <VMName> -DestinationStoragePath \\server\share\VMName


Methodology & Tools

9


Scenarios

Boot Linux/WinXP/Win2012 VM image Single host and multiple hosts Idle for 10 minutes

Build Linux Kernel Single host build and multiple host builds 1x vCPU and 4x vCPU

Random file operations Explorer and robocopy copies, small and

large files (4Kb – 1GB)

10


Methodology

1. Cold boot Win2012 SMB3 file server 2. Start packet capture (dumpcap.exe) 3. Cold boot Win2012 Hyper-V host 4. Launch scenario VM(s) 5. Cleanly shutdown scenario VM(s) 6. Stop packet capture 7. Shutdown Win2012 Hyper-V host 8. Shutdown Win2012 SMB3 File Server

11


Sharktools

Python extension library to allow using Wireshark’s packet dissector feature https://github.com/armenb/sharktools

Required small fixes for building against

Wireshark 1.10.1

Returns a dictionary of fields per frame { 'frame.number': 10,

'smb2.sesid': 2L, 'smb2.seq_num': 9589L, … } 12


Import pyshark

13

#!/usr/bin/env python import sys import pyshark pcap = pyshark.read( ‘/home/coffeedude/sample.pcap’, [ ‘frame.number’, ‘smb2.seq_num’ ], ‘smb2 and smb2.cmd == 0x05’) for f in pcap: print f {'frame.number': 10, 'smb2.seq_num': 9589L} {'frame.number': 101, 'smb2.seq_num': 9691L}


Dictionary Composition

14

SMB2 Request

SMB2 Response

SMB2 Request

SMB2 Response

SMB2 Request

SMB2 Response

SMB2 Request

SMB2 Response

OpExchange (Create)

File Handle f[name] = {

}


Scenario – Single Client Boot

15


Command Distribution

16

Linux WinXP Windows 2012 Command Run1 Run2 Run3 Run1 Run2 Run3 Run1 Run2 Run3 Negotiate 0.01% 0.01% 0.01% 0.01% 0.01% 0.01% 0.00% 0.00% 0.00%

SessionSetup 0.46% 0.49% 0.50% 0.45% 0.44% 0.46% 0.28% 0.27% 0.25% Logoff 0.44% 0.47% 0.48% 0.43% 0.42% 0.45% 0.27% 0.26% 0.25%

TreeConnect 0.47% 0.50% 0.51% 0.45% 0.45% 0.47% 0.29% 0.27% 0.26% TreeDisconnect 0.45% 0.48% 0.49% 0.44% 0.43% 0.46% 0.28% 0.27% 0.25%

Create 1.91% 2.04% 2.08% 1.67% 1.66% 1.75% 1.08% 1.03% 0.97% Close 0.51% 0.51% 0.57% 0.41% 0.46% 0.43% 0.27% 0.26% 0.25% Flush 0.36% 0.35% 0.34% 3.24% 3.34% 3.29% 1.39% 1.32% 1.29% Read 35.60% 32.61% 32.55% 38.09% 28.22% 29.31% 54.75% 54.94% 54.89%

Write 1.33% 1.60% 1.32% 14.20% 12.46% 13.43% 7.43% 7.56% 6.99% Lock 0.24% 0.26% 0.27% 0.12% 0.12% 0.13% 0.08% 0.07% 0.07% IoCtl 1.35% 1.44% 1.47% 1.14% 1.13% 1.19% 0.90% 1.00% 0.83%

Cancel 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Echo 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%

QueryDirectory 0.00% 0.00% 0.00% 0.02% 0.02% 0.02% 0.00% 0.00% 0.00% ChangeNotify 0.44% 0.47% 0.48% 0.38% 0.38% 0.40% 0.25% 0.24% 0.22%

QueryInfo 56.23% 58.55% 58.73% 38.84% 50.36% 48.10% 32.65% 32.45% 33.39% SetInfo 0.20% 0.21% 0.22% 0.11% 0.11% 0.11% 0.07% 0.07% 0.07%

OplockBreak 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%


Command Occurrences

17

Linux WinXP Windows 2012 Command Run1 Run2 Run3 Run1 Run2 Run3 Run1 Run2 Run3 Negotiate 1 1 1 1 1 1 1 1 1

SessionSetup 60 60 60 58 58 58 58 58 58 Logoff 58 58 58 56 56 56 56 56 56

TreeConnect 61 61 61 59 59 59 59 59 60 TreeDisconnect 59 59 59 57 57 57 57 57 58

Create 251 251 251 218 219 219 220 221 222 Close 67 63 69 53 61 54 55 56 57 Flush 47 43 41 422 442 412 284 284 295 Read 4668 4005 3926 4964 3730 3669 11170 11815 12526

Write 174 196 159 1851 1647 1681 1515 1626 1596 Lock 32 32 32 16 16 16 16 16 16 IoCtl 177 177 177 149 149 149 183 214 190

Cancel 0 0 0 0 0 0 0 0 0 Echo 0 0 0 0 0 0 0 0 0

QueryDirectory 0 0 0 2 2 2 0 0 0 ChangeNotify 58 58 58 50 50 50 51 51 51

QueryInfo 7373 7190 7085 5061 6657 6021 6660 6978 7619 SetInfo 26 26 26 14 14 14 15 15 15

OplockBreak 0 0 0 0 0 0 0 0 0

Total Requests 13112 12280 12063 13031 13218 12518 20400 21507 22820


VHDX Handle – Linux

18

Start Length Tags Access Mask Share Flags Create Options

1379126225.583680 15.566940 ['DH2Q' 'QFid' 'RqLs'] 0x120089 0x7 0x400060

1379126225.765800 15.385081 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48

1379126225.767010 15.384112 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48

1379126225.768160 15.383215 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48

1379126225.769270 15.382348 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48

1379126225.770370 643.326370 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48

1379126225.771510 15.380348 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40

1379126225.771860 0.083857 ['DH2Q' 'RqLs'] 0x120089 0x1 0x48

1379126225.852160 15.299951 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48

1379126225.855980 632.726568 ['DH2Q' 'RqLs'] 0x12019f 0x1 0x48

1379126284.873770 0.019414 0x100080 0x7 0x400060 … … … … … …

1379126859.659730 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x7 0x400060

1379126859.660920 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48

1379126859.662020 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48

1379126859.663040 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48

1379126859.664030 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 … … … … … …


VHDX Handle – Windows XP SP3

19

Start Length Tags Access Mask Share Flags Create Options 1378998568.137740 12.084596 ['DH2Q' 'QFid' 'RqLs'] 0x120089 0x7 0x400060 1378998568.238960 11.983569 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.239860 11.98279 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.240740 11.982038 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.241500 11.981468 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.242230 9999999999 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.242950 11.980128 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378998568.243260 0.071878 ['DH2Q' 'RqLs'] 0x120089 0x1 0x48 1378998568.312350 11.910827 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.315360 598.671185 ['DH2Q' 'RqLs'] 0x12019f 0x1 0x48 1378998627.382870 0.008148 0x100080 0x7 0x400060 1378998627.391410 12.39557 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998627.392170 12.395022 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998627.392950 12.394439 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998627.393670 12.393823 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998627.394340 12.393332 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378998978.645760 0.0013 0x100080 0x7 0x400060 1378998978.647370 11.562656 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998978.648080 11.562144 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998978.648790 11.56155 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998978.649480 11.56098 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998978.650150 11.560427 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378999166.986770 9999999999 ['DH2Q' 'RqLs'] 0x120089 0x1 0x48 1378999166.988020 9999999999 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48


VHDX Handle – Windows 2012

20

Start Length Tags Access Mask Share Flags Create Options 1378994455.071740 8.265029 ['DH2Q' 'QFid' 'RqLs'] 0x120089 0x7 0x400060 1378994455.242910 38.686086 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.244040 38.685176 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.244800 38.684545 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.245520 38.683969 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.246210 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.246910 38.682697 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378994455.247200 0.066620 ['DH2Q' 'RqLs'] 0x120089 0x1 0x48 1378994455.311260 38.618467 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.314040 617.598727 ['DH2Q' 'RqLs'] 0x12019f 0x1 0x48 1378994514.122400 0.007406 0x100080 0x7 0x400060 1378994514.130140 12.115971 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994514.130890 12.115480 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994514.131590 12.114990 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994514.132260 12.114522 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994514.132920 12.113976 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378994832.641740 0.000866 0x100080 0x7 0x400060 1378994832.642900 12.051110 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994832.643690 12.050523 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994832.644390 12.049935 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994832.645060 12.049385 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994832.645710 12.048850 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378995056.964460 18.050300 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378995056.965210 18.049824 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378995056.965970 18.049272 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378995072.912980 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x1 0x48 1378995072.914410 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48


0

20

40

60

80

100

120

140

VHDX – Write (Linux)

I/O Size - Linux (Ubuntu 12.04)

21

0

500

1000

1500

2000

2500

3000

VHDX – Read (Linux)


I/O Size - Windows XP SP3

22

0

500

1000

1500

2000

2500

3000

3500

4000

4500

Read - WinXP VHDX

0

200

400

600

800

1000

1200

512

1024

1536

2048

2560

3072

3584

4096

4608

5120

5632

6656

7168

8192

9216

9728

1075

212

288

1638

420

480

2457

628

672

3686

440

960

4505

653

248

5734

465

536

Write - WinXP VHDX


I/O Size - Windows Server 2012

23

0

500

1000

1500

2000

2500

3000

3500

4000

4500

512

2560

4608

6656

8704

1075

212

800

1484

816

896

1894

420

992

2304

025

088

2713

629

184

3123

233

280

3532

837

888

3993

642

496

4556

847

616

5017

653

760

5632

058

368

6041

662

976

6502

467

072

6963

271

680

7475

277

312

8345

687

040

9011

293

184

9728

010

2400

1080

3211

1616

1146

8812

2368

1305

60

Read – Win2012 VDHX


I/O Size - Windows Server 2012

24

0

100

200

300

400

500

600

700

800

Write – Win2012 VHDX


Definition: Jump Distance

25

Time Jump Distance Offset Read Size

1378952386.399140 0 0 4096

1378952386.411040 61440 65536 4096

1378952386.412700 61440 131072 4096

1378952386.413250 61440 196608 65536

1378952386.415470 1835008 2097152 65536

1378952386.425300 0 2162688 4096

1378952386.428070 978944 3145728 1048576

1378952386.538270 -4194304 0 512

1378952388.136860 37748224 37748736 32768


Jump – Linux (Ubuntu 12.04)

26

-5E+09

-4E+09

-3E+09

-2E+09

-1E+09

0

1E+09

2E+09

3E+09

4E+09

5E+09

Read Jump Distances: 0 – 633 sec

-5E+09

-4E+09

-3E+09

-2E+09

-1E+09

0

1E+09

2E+09

3E+09

4E+09

5E+09

Read Jump Distances: 0 – 30 sec



27

-5E+09

-4E+09

-3E+09

-2E+09

-1E+09

0

1E+09

2E+09

3E+09

4E+09

5E+09

Read Jump Distance: 22 – 30 sec



28

-4E+09

-3E+09

-2E+09

-1E+09

0

1E+09

2E+09

3E+09

4E+09

Write Jump Distance: 0 – 633 sec


Jump - Windows XP SP3

29

-2E+09

-1.5E+09

-1E+09

-5E+08

0

500000000

1E+09

1.5E+09

2E+09



Jump - Windows XP SP3

30

-2E+09

-1.5E+09

-1E+09

-5E+08

0

500000000

1E+09

1.5E+09

2E+09

Write Jump Distance: 0 – 140 sec


Jump - Windows Server 2012

31

-1E+10

-8E+09

-6E+09

-4E+09

-2E+09

0

2E+09

4E+09

6E+09

8E+09

1E+10



Jump - Windows Server 2012

32

-1E+10

-8E+09

-6E+09

-4E+09

-2E+09

0

2E+09

4E+09

6E+09

8E+09

1E+10

Write Jump Distance: 0 – 440sec


Boot – Single Host Summary

All reads / writes to the VHDX file are in multiples of 512 bytes Favorites are 4Kb, 32Kb, and 128Kb

Size and jump distance distribution changes with

guest OS

Multiple persistent handles (4 or more) opened per VM instance; only one opened for write access 33


Scenario – Kernel Compile

Linux

34


Kernel Build Script

35

#!/bin/bash scp -p nas1:~/tmp/linux-3.10.4.tar . tar xf linux-3.10.4.tar cd linux-3.10.4 make defconfig && make -j2 all cd .. /bin/rm -rf linux-3.10.4* sync shutdown -h now


36

1x vCPU (1080 sec) 4x vCPU (360 sec) Command Run1 Run2 Run1 Run2 Run3 Negotiate 0.00% 0.00% 0.00% 0.00% 0.00%

SessionSetup 0.22% 0.23% 0.19% 0.28% 0.16% Logoff 0.21% 0.22% 0.18% 0.27% 0.16%

TreeConnect 0.22% 0.23% 0.19% 0.28% 0.16% TreeDisconnect 0.22% 0.22% 0.19% 0.27% 0.16%

Create 0.80% 0.80% 0.66% 0.96% 0.58% Close 0.18% 0.17% 0.12% 0.17% 0.12% Flush 0.94% 0.63% 0.45% 0.26% 0.43% Read 34.88% 34.25% 37.87% 44.77% 43.79%

Write 36.53% 44.25% 46.89% 42.91% 48.96% Lock 0.04% 0.04% 0.03% 0.05% 0.04% IoCtl 0.55% 0.56% 0.45% 0.67% 0.40%

Cancel 0.00% 0.00% 0.00% 0.00% 0.00% Echo 0.00% 0.00% 0.00% 0.00% 0.00%

QueryDirectory 0.00% 0.00% 0.00% 0.00% 0.00% ChangeNotify 0.18% 0.19% 0.16% 0.23% 0.14%

QueryInfo 24.74% 18.17% 12.58% 8.84% 4.85% SetInfo 0.28% 0.03% 0.03% 0.04% 0.05%

OplockBreak 0.00% 0.00% 0.00% 0.00% 0.00%



37

1x vCPU (1080 sec) 4x vCPU (360 sec) Command Run1 Run2 Run1 Run2 Run3 Negotiate 1 1 1 1 1

SessionSetup 60 60 60 60 60 Logoff 58 58 58 58 58

TreeConnect 61 61 61 61 61 TreeDisconnect 59 59 59 59 59

Create 218 213 207 205 215 Close 49 44 38 36 44 Flush 255 168 143 55 160 Read 9461 9070 11929 9609 16304

Write 9909 11718 14769 9210 18229 Lock 10 10 10 10 16 IoCtl 149 149 143 143 148

Cancel 0 0 0 0 0 Echo 0 0 0 0 0

QueryDirectory 0 0 0 0 0 ChangeNotify 50 50 50 50 53

QueryInfo 6711 4813 3962 1898 1805 SetInfo 75 9 9 9 17

OplockBreak 0 0 0 0 0

Total Requests 27126 26483 31499 21464 37230

Command Occurrence


I/O Size – Kernel Build (4x vCPU)

38

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

VHDX Read Sizes


I/O Size – Kernel Build (4x vCPU)

39

0

1000

2000

3000

4000

5000

6000

VHDX Write Sizes


Jump – Kernel Build (4x vCPU)

40

-6E+09

-4E+09

-2E+09

0

2E+09

4E+09

6E+09

Read jump Distance: 0 – 360 sec


Jump – Kernel Build (4x vCPU)

41

-6E+09

-4E+09

-2E+09

0

2E+09

4E+09

6E+09

Chart Title


Scenario – Random Tree Copy

Windows Server 2012

42


Scenario Details

Guest using SCSI virtual disk Tree copy of deep directory tree 5.8 Gb of allocated space 25,000 files 2,500 directories

Concurrent copies using explorer.exe robocopy.exe

43


44


Command Occurrences Negotiate 0 0.00%

SessionSetup 1 0.00% Logoff 0 0.00%

TreeConnect 2 0.00% TreeDisconnect 1 0.00%

Create 55 0.07% Close 52 0.06% Flush 1089 1.30% Read 51900 61.95%

Write 26789 31.98% Lock 19 0.02% IoCtl 2653 3.17%

Cancel 0 0.00% Echo 0 0.00%

QueryDirectory 0 0.00% ChangeNotify 4 0.00%

QueryInfo 1131 1.35% SetInfo 78 0.09%

OplockBreak 0 0.00%

Total Requests 83774


I/O Size – Random Tree Copy

45

0

500

1000

1500

2000

2500

3000

512 1024 2048 4096 8192 12288 16384 20480 31232 32768 65536 122880 131072 917504 925696 1048576

SCSI VHDX Read Sizes


I/O Size – Random Tree Copy

46

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

4096

2048

036

864

5324

869

632

8601

610

2400

1187

8413

5168

1515

5216

7936

1843

2020

0704

2170

8823

3472

2498

5626

6240

2826

2429

9008

3153

9233

1776

3481

6036

4544

3809

2839

7312

4136

9643

0080

4464

6446

6944

4833

2850

7904

5242

8854

0672

5570

5657

3440

5980

1661

8496

6430

7266

7648

6922

2470

8608

7290

8874

5472

7659

5278

6432

8110

0883

1488

8519

6887

2448

9093

1293

3888

9707

5210

0352

010

1990

410

3628

8

SCSI VHDX Write Sizes


Jump – Random Tree Copy

47

-3E+09

-2E+09

-1E+09

0

1E+09

2E+09

3E+09

Write Jump Distance


Closing Thoughts

48


Conclusions

SMB3 command distribution shows heavy read/write but with large QueryFileInfo calls to the machine’s XML config

Hyper-V commonly uses larger SMB2 I/O sizes than, but LargeMTU does not appear critical

Pattern shows multiple long lived file handles per guest instance

Read and Write requests are not interleaved on the same handle

49


Future Work

(Re-)expand scope to cover SMBDirect, Multi-Channel, and Failover

Expand to include more guest workloads Apache, MS-SQL Server, etc…

Expand to include other generalized SMB3

workloads Home directories

50


Questions or Comments?

Gerald Carter [email protected]

EMC/Isilon

51

Documents

An SMB3 Engineer’s View of Windows Server 2012 Hyper-V ......Windows 2012: Command Run1 Run2 Run3 Run1 Run2 Run3 Run1 Run2 Run3 Negotiate 1 1 1 1 1 1 1 1 1 SessionSetup 60 60 60