Simplify your Software Testing Process · The performance is nearly the same compared to SoC's thanks to X86 architecture Don’t need to specify the hardware like the board or peripherals

Simplify your Software Testing Process

Fuzzing of Embedded Systems

Table of content

● What the Hack is Fuzzing ?

● Approaches to fuzz embedded Projects

● Improve Fuzzer Inputs with Grammar Mutation

2

The speakers

3

Sirko HöerB.Sc.

Christian HartlageB.Sc.

● Vulnerability Research in Embedded

Systems at Code Intelligence GmbH

● More than 10 years of experience in

the area of Cyber Security

● [email protected]

● Development of new methods to

improve fuzzing coverage at

Code Intelligence GmbH

● More than 3 years of experience with

automated malware analysis at FKIE

● [email protected]

Overview

4

What is Fuzzing ?

Overview

5

What is Fuzzing in theory ?

Fuzzing or fuzz testing is an automated software testing technique that involves random data as input to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks

Overview

6


Fuzzing isor fuzz testing is an automated stuffing testing technique that involves random data into a computer program until ogram is then monitored until it has crashes, failing built-in code assertions, or potential memory leaks

Overview

7


Fuzz-Target

Overview

8


Fuzz-TargetFuzz-Engine

Overview

9


Fuzz-TargetFuzz-EngineRandom Input

● seed ● mutation / mixing / permutation

Monitoring

Overview

10


Fuzz-TargetFuzz-Engine CRASHRandom Input

● Memory Leak ● Crashes● Failing built-in code assertions● Racing Conditions

Monitoring

Overview

11

Fuzzing

Blackbox Whitebox / Greybox

Out-of-Process FuzzingIn-Process Fuzzing...Protocol Fuzzer

Types of Fuzzing techniques

Overview

12

Fuzzing

Blackbox Whitebox / Greybox

Out-of-Process FuzzingIn-Process Fuzzing...Protocol Fuzzer

Types of Fuzzing techniques

What is coverage-guided Fuzzing ?

13

What does coverage mean?

Engine

libFuzzer

MutatorFeedback

handler

Fuzz-Target

IO API Modul A

Modul BModul CSeed


14


Engine

libFuzzer

MutatorFeedback

handler

Fuzz-Target

IO API Modul A

Modul BModul C

“s1t3\x8f\x44”

Seed


15


Engine

libFuzzer

MutatorFeedback

handler

Fuzz-Target

IO API Modul A

Modul BModul C

“s1t3\x8f\x44”

Seed


16


Engine

libFuzzer

MutatorFeedback

handler

Fuzz-Target

IO API Modul A

Modul BModul CSeed

Coverage information

mutated Inputs


17

Which coverage types are available ?

● Line coverage

● Function coverage

● Data Flow coverage

● Custom Definitions (eg. malloc / free)

● Edge coverage

VUzzer AFL libFuzzer honggfuzz

Assembler Rewrite ✓ ✓

Compile Time ✓ ✓ ✓

Hardware (Intel BTS/PT) ✓

i.e. : -fsanitize-coverage=edge,trace-pc,trace-cmp

Challenges to make Fuzzing possible with libFuzzer

18

● needs llvm, clang and the compiler-rt

● needs knowledge, how to write a Fuzz-Target

● it is complicated to embed it into a build environment

What do you need to run libFuzzer

[Source]: https://en.wikipedia.org/wiki/LLVM

Challenges to make Fuzzing possible with libFuzzer

19

● compile target project with specific options

like -fsanitize-coverage=*, -fsanitize=address, ...

● write a Fuzz-Target like a Unit Test

● compile the Fuzz-Target and link against the

target project

● then we get a executable

Steps to run fuzz-tests with libFuzzer

FUZZ-Target

Target Project

compiler-rt

FuzzerSeed

Approaches to fuzz embedded Projects

20

What is the Problem ?

Problem statement

● Need to cross-compile the code

● Hardware limitations

● Many IoT devices have their own operating system

Fuzz Engine

21[1] Sources: https://tr1.cbsistatic.com/hub/i/2016/04/18/34fe8362-ee13-4785-9e2c-0aa26740dcda/iothero.jpg

[1]

Problem statement

● Need to cross-compile the code

● Hardware limitations

● Many IoT devices have their own operating system

Limitation of available tools

● Hard to setup the environment

● libFuzzer on Linux for ARM-Architecture (32 / 64 Bit)

not officially supported and available ● Sanitizer on Linux for ARM-Architecture (32 / 64 Bit)

not officially supported and available ● also for MIPS (32 / 64 Bit)

[1] Sources: https://tr1.cbsistatic.com/hub/i/2016/04/18/34fe8362-ee13-4785-9e2c-0aa26740dcda/iothero.jpg

Fuzz Engine

22

[1]

Corpus

Our approach

[1] Sources: https://github.com/google/sanitizers/wiki/AddressSanitizer23

Modul Tests Integration Tests System Tests

Modul 1 Modul n

API

Modul 1 Modul n

Corpus

Our approach



Modul 1 Modul n

Fuzzing

API

Modul 1 Modul n

Fuzzing Fuzzing Fuzzing

Corpus

Our approach



Modul 1 Modul n

Fuzzing

API

Modul 1 Modul n

FuzzingFuzzing Fuzzing Fuzzing

used as seed

Corpus

Our approach



Modul 1 Modul n

Fuzzing

API

Modul 1 Modul n


used as seed used as seed

Black box Fuzzing

Corpus

Our approach



Modul 1 Modul n

Fuzzing

API

Modul 1 Modul n


Black box Fuzzing


Virtualization with QEMU Hardware

Corpus

Our approach



Modul 1 Modul n

Fuzzing

API

Modul 1 Modul n




Black box Fuzzing

What is QEMU ?

29

● Quick Emulation (QEMU)

● Developed by Fabrice Bellard in 2003

● System level emulation

● Binary translation

● is an open source project

Features :

- CPU emulator

- Emulated devices (e.g. VGA Display, serial port,...)

- Generic devices (e.g. block devices, network devices,...)

- Machine descriptions (e.g. PC, PowerMac,...)

- Debugger

- User interface

Some notations

● Host architecture : where you run QEMU

● Target architecture : the emulated architecture

[Source]: https://wiki.qemu.org/Logo

Features :

● it can be used for unit - and integration tests

● has a high performance

● very good scalability !

but …

● Needs a cross compile environment

● Needs to adapt libFuzzer and clang

Fuzzing with QEMU User Mode

30


31

Workflow to make fuzzing possible in QEMU User Mode

● build llvm, clang and compiler-rt

● build Fuzz-Target● be aware of all

dependencies

● install and setup Cross-Compile environment

● available on all frequently used linux distributions

● i.e. Debian/Ubuntu, CentOS, Arch Linux

Get X-Compile Env Build llvm toolchain Build Fuzz-Target Run fuzzer

● run Fuzz-Target ● use QEMU binary

translation

● Fuzzer run natively on the (X86-)Host

Modify libFuzzer for embedded Projects

Modification of compiler-rt sources for arm :

1. change some files of build configs

2. Add new options for libFuzzer

a. --qemu_user_mode

b. --qemu_toolchain_lib

3. Implement the functionality

32

Example :

[user@host] $ qemu-arm -L /usr/arm-none-gnueabihf ./fuzz_target \

--qemu_user_mode=1 --worker=4 --jobs=4 corpus

Generate command and execute it multiple times depending the number of jobs “qemu-arm -L <path> fuzz_target corpus > fuzz-0.log 2>&1”

qemu-arm -L /usr/arm-linux-gnueabihf fuzz_target corpus >fuzz-0.log 2>&1




Modify libFuzzer for ARM

Modification of compiler-rt sources for arm :

1. change some files of build configs

2. Add new options for libFuzzer

a. --qemu_user_mode

b. --qemu_toolchain_lib

3. Implement the functionality

33

Example :

[user@host] $ qemu-arm -L /usr/arm-none-gnueabihf ./fuzz_target \

--qemu_user_mode=1 --worker=4 --jobs=4 corpus

Generate command and execute it multiple times depending the number of jobs “qemu-arm -L <path> fuzz_target corpus > fuzz-0.log 2>&1”





[Source]: https://i.imgflip.com/x2cxg.jpg

Modify libFuzzer for ARM

Build example :

[user@host] $ clang --target arm-none-gnueabihf -L /usr/arm-none-gnueabihf/lib \

--sysroot= /usr/arm-none-gnueabihf -I /usr/arm-none-gnueabihf/include \

--resources /usr/arm-none-gnueabihf/lib/clang/7.0.0/linux/lib example.c

Steps to get llvm for arm :

● Setup X-Compile environment (arm-none-eabi, arm-linux-eabi, …)

● Modify libFuzzer Sources, such that will compile into ARM architecture

● Install the new binary to the specific path

● specify the path with compile option --resources

34

Advantages :

● You don’t need specific devices

● Don’t need to install a full system (it saves time)

● The performance is nearly the same compared to SoC's thanks to X86 architecture

● Don’t need to specify the hardware like the board or peripherals

● Easy to automate the build process for example with docker

● no RAM limitations, scalable


Disadvantages :

● Need a cross compile environment, but only once !

● performance weakness (?)

35

Corpus

Our approach



Modul 1 Modul n

Fuzzing

API

Modul 1 Modul n


Fuzzing



Features :

● complete system test of the project

● simulation of external interfaces like bluetooth, CAN-Bus or others

but …

● testing has slow performance (low executions per seconds)

● only a limited number of boards available !!

● implementing new or custom boards needs a lot of time

Fuzzing with QEMU full system emulation

37

How does it work :

● You have to figure out which board you use for fuzzing

● Depending on the board, there are specific processors available

○ Example : versatileab

- arm1026

- arm1136 ...

● You have to choose peripheral like the network card or a hard disk drive

● You have to download the right (netinstall)-kernel to start the system and maybe install a new OS

● Compiling projects inside the virtual machine is the same as on a real machine [2]


Example :

[user@host] $ qemu-system-arm -M versatileab -cpu arm1136 -hda disk.vdi -append "root=/dev/sda2 rootfstype=ext4 rw" -serial stdio

38

Advantages :

● Can be used for acceptance test

● QEMU covers for almost all main architectures

● Changing hardware settings of the virtual board is easy (add CPU cores, change RAM-size, ...)

● Opportunity to fuzz kernel code


Disadvantages :

● Hard to setup a virtual environment, it takes time

● Very hard to automate the setup process as well as the fuzzing process

● performance weakness (more in the evaluation)

39

[Source]: https://wiki.qemu.org/Logo

Corpus

Our approach



Modul 1 Modul n

Fuzzing

API

Modul 1 Modul n


Fuzzing



Advantages :

● Easy to setup the system environment

● Applications and binaries runs on a nativ system / real interfaces

Disadvantages :

● Uses black box fuzzing on protocols

● Doesn’t have the feedback technique like coverage-guided fuzzing

● there are more than thousands devices devices on the market available

● limitations of the SoC boards (performance is weak)

Fuzzing on SoC devices

41

[Source]: https://cdn-reichelt.de/bilder/web/xxl_ws/A300/RASP_03_01.png

42

What does the performance look like ?

Test setup and projects

● Compare the performance between :○ SoC devices○ Virtual machines on two boards○ QEMU user mode on my laptop

● Metrics : ○ Executions per seconds○ Memory usage

SystemSoC

Board(RPi)

Vexpress-A9 virt Laptop

Architecture ARMv7 ARMv7 ARMv7 X86_64

CPU-Cores4

(Cortex-A9)

1(Cortex-

A9)

4(Cortex-

A53)

8(Core i7)

RAM 1 BG 1 GB 4 GB 32 GB

OS ArchLinux Debian Debian ArchLinux

Projects :

● RE2 - RegEx parser written in c++

● woff2 - WOFF File Format 2.0

● These projects belong to the fuzzer-test-suite

from google[https://github.com/google/fuzzer-test-suite]

43

Evaluation of RE2

System SoC Board(RPi) Vexpress-A9 virt QEMU mode

exec / s(after 1 min) ~6500 ~980 ~850 ~4000

exec / s(after 5 min) ~5000 ~1010 ~980 ~2900

exec / s(after 10 min) ~3200 ~830 ~800 ~1700

exec / s(after 60 min) aborted aborted ~180 ~460

time

44

Evaluation of WOFF

System SoC Board(RPi) Vexpress-A9 virt QEMU mode

exec / s(after 1 min) ~37000 ~6000 ~17000 ~35000

exec / s(after 5 min) ~33000 ~5800 ~13000 ~30000

exec / s(after 10 min) ~9000 ~5000 ~2500 ~15000

exec / s(after 60 min) ~5000 ~2000 ~2000 bug found

time

45

● QEMU-Mode has the best long term performance

● Full system emulation is nearly 80 % slower than the SoC

● The longer the test goes the slower the execution speed is

● Fuzzing on SoC is limited by the memory and the processor

Summary of performance evaluation

46

47

Structure Aware FuzzingOr how to stop wasting 99.9% of your Fuzzer

Runs

48

Why make the fuzzer structure-aware?

49

50

51

{ "name": "John Wick", "job": "Assassin", "pets": ["Buster"]}

52

{ "name": "John", "pets": ["Buster"}}






{ "name": "John Wick", "job": "Assassin", "pets": ["Buster"}}

53







{ "name": "John Wick", "job": "Assassin", "pets": ["Buster"}}

54

How to make the fuzzer structure-aware?

55

“We at Google have a big Hammer. It’s called Protobuf”

-- Kostya Serebryany, Google LibFuzzer Team

Protobuf Example

56

message Person { string name = 1; string profession = 2;

message Pet { string name = 1; }

repeated Pet pets = 3;}

person.proto

#include <stdio.h>#include "person.pb.h"

int main() { Person p; p.set_name("John Wick"); p.set_profession("Assassin"); p.add_pets().set_name("Buster"); p.add_pets().set_name("Spot"); std::cerr << p.DebugString();}

person.cpp

name: "John Wick"profession: "Assassin"

pets { name: "Buster"}

pets { name: "Spot"}

stdout

57

message JsonValue { oneof value_oneof { JsonObject object = 1; JsonArray array = 2; bytes string = 3; Number number = 4; bool boolean = 5; }}message JsonObject { repeated JsonKeyValPair keyvalpairs = 1;}message JsonArray { repeated JsonValue value = 1;}message JsonKeyValPair { required bytes key = 1;}message Number { required bool scientific = 1 [default = false]; oneof number_oneof { sint32 number = 2; double float = 3; }}

...[Source]: https://www.json.org/

Protobuf Conversion

58

{ "name": "John Wick", "profession": "Assassin", "pets": ["Buster"]}

object { keyvalpairs { key: "name" value { string: "John Wick" } } keyvalpairs { key: "profession" value { string: "Assassin" } } keyvalpairs { key: "pets" value { array { value { string: "Buster" } } } }}

ProtoToJson()

JsonToProto()

Protobuf Conversion

59

name: John Wickprofession: Assassinpets: - Buster - Spot

object { keyvalpairs { key: "name" value { string: "John Wick" } } keyvalpairs { key: "profession" value { string: "Assassin" } } keyvalpairs { key: "pets" value { array { value { string: "Buster" } } } }}

ProtoToYaml()

YamlToProto()

Protobuf Mutation

60

https://github.com/google/libprotobuf-mutator

[Source]:https://www.reddit.com/r/utterlyuseless/

Protobuf Mutation

● Protobuf is strict -> every mutation is again valid protobuf

● Every valid Protobuf can be converted to valid JSON/YAML/XML/…

● Mutation algorithm can be customized with little effort

● Integration into libFuzzer (!)

● Adding a new structured language is only 3 steps

○ Writing a Proto Message Definition

○ Write a Converter to Proto

○ Write a Converter to Structured Language

● You’re ready to fuzz only in the language you defined!

61

Protobuf Mutation

● Message Definition in Proto is higher level abstraction

●

62

Real-life Evaluation

● We took a closer look at GnuPG Made Easy (GPGME)

● GPGME ships own JSON parser

63

● Sergej Dechand and Sirko Höer (both Code Intelligence) found 12 bugs

● 2 Stackoverflows

● also integer overflows and memory leaks

Conclusion

64

● Three different approaches for embedded fuzzing are shown

● Modify compiler-RT to make libFuzzer available for ARM32/ ARM64 / MIPS32 / MIPS64

● The performance of QEMU-User-Mode is pretty impressive (partly much faster than SoC)

● Scalable : it can be executed in a docker container on a server

● Structure Aware Fuzzing is able to minimize Fuzzer Runs on parsing engines

● Bypasses Parsers, enables fuzzing of the business logic

65

Are there any questions left ?

66

Thank you for your attention

Documents

Simplify your Software Testing Process · The performance is nearly the same compared to SoC's thanks to X86 architecture Don’t need to specify the hardware like the board or peripherals