FFRI,Inc.

Fourteenforty Research Institute, Inc.

FFRI,Inc. http://www.ffri.jp

Automated on-execute test using VirtualBox

Junichi Murakami Executive Officer, Director of Advanced Development Division

Ver2.00.01

FFRI,Inc.

1. Background and motivation

2. Overview of a test

– automated on-execute test

– virtualization software and automation methods

– Oracle VM VirtualBox and its automation

– example of VBoxManage

3. Automation script

– FFRI AutoMonkey

– design concept

– throughput

– performance

4. References

5. Contact information

Agenda

2

FFRI,Inc.

• Automated test against a large amount of malware is required to evaluate a malware detection engine

• Testing methods are classified into on-demand and on-access testing

• on-execute test which is a kind of on-access test has to execute malware one by one

• Therefore automation based on virtualization is required

• This slides describes automated on-execute test method using VirtualBox

1.Background and motivation

3

malware scanning on-demand

on-access on-read/write

on-execute

* type of malware scanning

FFRI,Inc.

• Basic steps are following

1. Copy malware into a guest

2. Execute copied malware in the guest

3. Analyze or detect malware in the guest

4. Preserve the result after execution is terminated

5. Revert the guest back to original condition

6. Go to 1.

• Required functions to execute above are following

a. Copying a file to a guest from a host (copy-to)

b. Executing arbitrary a program in a guest from a host(exec)

c. Copying a file from guest to a host (copy-from)

d. Reverting a guest condition based on a snapshot(revert)

2.1.Automated on-execute testing

4

→ All functions can be achieved by making a communication interface between a host

and a guest using TCP/IP. We considered the way we do not need to involve

developing software as possible as we could

FFRI,Inc.

software Licence copy-to copy-from exec revert method

VMware Workstation Proprietary ○ ○ ○ ○ VIX API

VMware ESX(#1) Proprietary

○ ○ ○ ○ VIX API

Oracle VM VirtualBox GPL2 ○ ○ ○ ○ VBoxManage

QEMU + KVM GPL2(#2) × × × ○ Libvirt

2.2.virtualization software and automation methods

5

#1 ESXi can also use VIX API for 60days by registering a evaluation license.

#2 KVM's parts are licensed under various GNU licenses(GPL, GPL2, LGPL2, etc.)

• Use functions which virtualization software has natively

• VMware(licensed) and VritualBox have all the features we need → We considered using VirtualBox because of the cost advantage

• QEMU+KVM can be used by 3rd party software(ex: libguestfs + winexe)

– “Malware Analysis: Collaboration, Automation & Tuning”, Shmoocon 2013 http://www.slideshare.net/xabean/malware-analysis-16674048

FFRI,Inc.

• A kind of x86 virtualization software, currently developed by Oracle

• Version 4.0 and later, fully open source software (GPL2)

• Supporting various host and guest environments

– HostOS：Windows, Linux, Mac OS X, Solaris

– GuestOS：Windows, Linux, FreeBSD, OpenBSD, Mac OS X Server, Solaris,etc.

• CLI is available (VBoxManage), friendly to automation

– startvm , pause, resume, poweroff, clonevm, showvinfo

– copyto, copyfrom, exec

– taking snapshot and reverting

– control virtual machine devices status, etc.

2.3.Oracle VM VirtualBox and its automation

6

FFRI,Inc.

2.4.Example of VBoxManage

7

% vboxmanage startvm vm

% vboxmanage controlvm vm poweroff

% vboxmanage snapshot vm restore snapshot-1

% vboxmanage guestcontrol exec vm --image “c:/windows/system32/calc.exe” ¥

--username admin --timeout 60000 --wait-exit

% vboxmanage guestcontrol vm copyto “/some/file” “c:/file.txt” --username admin

* starting a guest

* power off a guest

* reverting a guest based on a snapshot

* execute a program in a guest from a host

* copying a file to a guest from a host

FFRI,Inc.

• Automation script using VBoxManage, just a shell script

– auto-monkey.sh:automation for copy, exec, copy, revert steps

– watch-monkey.sh：watch dog script for the monkey

• It can execute multiple test simultaneously, works individually

• Published at our website below, see README for the detail (License: BSD)

– http://www.ffri.jp/research/freeware.htm

3.1.FFRI AutoMonkey

8

VirtualBox(guest)

malware

log

auto-monkey.sh

watch-monkey.sh

VirtualBox(guest)

malware

log

auto-monkey.sh

watch-monkey.sh

image

snapshot

image

snapshot

FFRI,Inc.

• conform to KISS principle

• Estimation of remaining time is important for this kind of test

– we cannot determine when it would finish if the script hangs up

• Stability of VBoxManage (and VIX API) is the lifeline for the automation

• In fact, error occurs when it runs long time

– Failure by error

• exits immediately

• resumed a test automatically by watch-monkey.sh

– Hanging up(stuck) by error

• watch-monkey.sh monitors lifetime of a VirtualBox process

• if it is stuck, kill and resume

3.2.Design concept

9

FFRI,Inc.

• Testing under 1host and 7guest environment

• Processed 20,000 malware, each execution time was 60 seconds

– total elapsed time: 37h15m

– throughput：8.95 malware/minute # if malware execution terminated less than 60 seconds, the script processes next item.

• Host and guest environment is following

3.3.Throughput

10

Hardware CPU: Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz

Memory: 8GB

HDD: 1.8TB x 1

Host OS Ubuntu 13.04 + VirtualBox 4.2

Guest OS Windows XP SP3(x86) + FFR yarai 2.3

CPU:1 CPU

Memory:750MB

FFRI,Inc.

• About 70% of total processor is idle state(each core also indicates same trend)

3.4.performance - processor

11

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0:0

5

0:5

5

1:4

5

2:3

5

3:2

5

4:1

5

5:0

5

5:5

5

6:4

5

7:3

5

8:2

5

9:1

5

10

:05

10

:55

11

:45

12

:35

13

:25

14

:15

15

:05

15

:55

16

:45

17

:35

18

:25

19

:15

20

:05

20

:55

21

:45

22

:35

23

:25

load

all processsor statistics

%idle

%steal

%iowait

%system

%nice

%user

FFRI,Inc.

• Consuming about 80% - 90% memory steadily

– real memory usage is between 2.5 and 4.0GB

3.4.performance - memory

12

0

500000

1000000

1500000

2000000

2500000

3000000

3500000

4000000

4500000

5000000

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0:0

5

1:0

5

2:0

5

3:0

5

4:0

5

5:0

5

6:0

5

7:0

5

8:0

5

9:0

5

10

:05

11

:05

12

:05

13

:05

14

:05

15

:05

16

:05

17

:05

18

:05

19

:05

20

:05

21

:05

22

:05

23

:05

memory utilization statistics

kbmemused

kbmemfree

kbmemused-(kbbuffers+kbcached)

FFRI,Inc.

• Disk busy ratio(%util) stays around 30% steadily

• The number of queued requests is between 4 and 8

3.4.performance – Disk IO

13

0

10

20

30

40

50

60

70

80

90

0

2

4

6

8

10

12

0:0

5

0:5

5

1:4

5

2:3

5

3:2

5

4:1

5

5:0

5

5:5

5

6:4

5

7:3

5

8:2

5

9:1

5

10

:05

10

:55

11

:45

12

:35

13

:25

14

:15

15

:05

15

:55

16

:45

17

:35

18

:25

19

:15

20

:05

20

:55

21

:45

22

:35

23

:25

0:0

0

Usage of block device(/dev/sda)

avgqu-sz

%util

FFRI,Inc.

• None of CPU, memory and IO wasn’t bottleneck under 1host and 7guest environment

• It seems we can add some more guests up to around 10 VMs according to memory usage

• However, we have to consider requirement of a process which is executed in a guest (cpu, memory)

3.4.performance - consideration

14

FFRI,Inc.

• http://www.ffri.jp/assets/files/research/freeware/FFRIAutoMonkey-1.0.tgz

• https://www.virtualbox.org/manual/UserManual.html

• http://www.slideshare.net/xabean/malware-analysis-16674048

• http://www.youtube.com/watch?v=peHdyUlchSM

• http://libguestfs.org/

• http://sourceforge.net/projects/winexe/files/

References

15

FFRI,Inc.

• E-Mail

– research-feedback@ffri.jp

• Twitter

– @FFRI_Research

Contact Information

16