Policies and Mechanisms for Operating System Securityvg/talks/os-integrity/osi.pdf · Example 2:...

Policies and Mechanisms for

Operating System Security

Vinod Ganapathy

vinodg@cs.rutgers.edu

Associate Professor of Computer Science

Rutgers, The State University of New Jersey

Layered computer system design

Modern computer systems are built using

layers of abstraction

Memory I/O devicesCPU

Hardware

Vinod Ganapathy - Policies and Mechanisms for OS Security

Hardware

Operating

System

SyscallsProcess

Kernel

Hardware

Operating

System

SyscallsProcess

Kernel

Utilities &

Librariesls, ps, &

bash utilitieslibc gcc …

Hardware

Operating

System

SyscallsProcess

Kernel

Utilities &

Librariesls, ps, &

Fundamental principle in security

Hardware

Operating

System

SyscallsProcess

Kernel

Utilities &

Librariesls, ps, &

The lower you go, the more control you have

Least control

Most control

Hardware

Operating

System

Utilities &

Libraries

Example: Malware detection

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

cat ps ls …TCB

Trusted

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

But utilities may be compromised!

cat ps ls

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

cat ps ls

1 Show me

file contents

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

cat ps ls

1 Show me

file contents

2 Fake, benign

content

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

Solution: Query the OS

System call API

Query with syscall

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

System call API

Query with syscall

OS reads file

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

System call API

Query with syscall

OS reads file

Returns true

file content

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

OS detects malicious utilities too

System call API

cat file

Read file

A Bdiff vs ?

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

What if the OS is malicious?

System call API

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

Rootkit = Malware that infects OS

System call API

Rootkits hide malware from detectors Long-term stealth

How does an OS get infected?

• Exploits of kernel vulnerabilities:

– Injecting malicious code by exploiting a memory

error in the kernel

• Privilege escalation attacks:

– Exploit a root process and use resulting

administrative privileges to update the kernel

• Social engineering attacks:

– Trick user into installing fake kernel updates

• Defeated via signature verification of kernel updates

• Trivial to perform prior to the Windows Vista OS

Vinod Ganapathy - Policies and Mechanisms for OS Security 19

How prevalent are rootkits?

• 2010 Microsoft report: 7% of all infections

from client machines due to rootkits[1]

• 2016 HummingBad Android rootkit:[2]

– Up to 85 million Android devices infected?

– Earns malware authors $300,000 each week

through fraudulent mobile advertisements

• Used in many high-profile incidents:

– Torpig and Storm botnets

– Sony BMG (2005), Greek wiretapping (2004/5)

[1] Microsoft Malware Protection Center, “Some Observations on Rootkits,” January 2010,

https://blogs.technet.microsoft.com/mmpc/2010/01/07/some-observations-on-rootkits

[2] CheckPoint Software, “From HummingBad to Worse,” July 2016,

http://blog.checkpoint.com/wp-content/uploads/2016/07/HummingBad-Research-report_FINAL-62916.pdf

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

How can we detect rootkits?

System call API

Hypervisor (a.k.a. Virtual Machine Monitor)

Ask for help from the layers below

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

How low can we go?

Hypervisor [Bluepill, Subvert]

Operating

System

Malware

detector

Utilities &

Libraries

How low can we go?

Hardware[Stuxnet, Trojaned ICs]???

TCBVinod Ganapathy - Policies and Mechanisms for OS Security

Today’s talk

• Detecting OS-level rootkit

infections (with some help

from the hardware)

• In two parts:

1. Policies: How do we know

that the OS is infected?

2. Mechanisms: How can the

hardware help us?

Utilities &

Libraries

Operating

System

HardwareTCB

My contributions to OS security

2008 ---- Detecting rootkits using data structure invariants [ACSAC’08]

2008 ---- Re-architecting device drivers for better isolation [ASPLOS’08]

2009 ---- Securing OSes from malicious device drivers [ACSAC’09]

2010 ---- Exploring rootkits on smartphones [HotMobile’10]

2011 ---- Security/energy tradeoffs in rootkit detection [ACM MobiSys’11]

2012 ---- Rootkit detection on cloud platforms [ACM CCS’12]

2013 ---- Adapting multicore hardware for rootkit detection [TIFS’13]

2016 ---- Rootkit detection with ARM TrustZone [ACM MobiSys’16]

2016 ---- Rootkit detection using 3D-stacked hardware [Submitted]

Covered in today’s talk

2008 ---- Detecting rootkits using data structure invariants [ACSAC’08]

2008 ---- Re-architecting device drivers for better isolation [ASPLOS’08]

2009 ---- Securing OSes from malicious device drivers [ACSAC’09]

2010 ---- Exploring rootkits on smartphones [HotMobile’10]

2011 ---- Security/energy tradeoffs in rootkit detection [ACM MobiSys’11]

2012 ---- Rootkit detection on cloud platforms [ACM CCS’12]

2013 ---- Adapting multicore hardware for rootkit detection [TIFS’13]

2016 ---- Rootkit detection with ARM TrustZone [ACM MobiSys’16]

2016 ---- Rootkit detection using 3D-stacked hardware [Submitted]

Modus operandi

Analysis of memory snapshots obtained from target machine

User app

Physical Memory

Hardware

Operating

System

SyscallProcess

Kernel

User app

Utilities &

Libraries

Target machinePotentially rootkit infected

Modus operandi

User app

Physical Memory

Hardware

Operating

System

SyscallProcess

Kernel

User app

Utilities &

Libraries

Analysis machineTrusted

Modus operandi

User app

Physical Memory

Hardware

Operating

System

SyscallProcess

Kernel

User app

Utilities &

Libraries

Analysis machineTrusted

Snapshot of

memory pages

Research questions

• RQ1: What algorithm should we use for memory snapshot analysis?

– Concerns our security policy

– Answer: Formulate rootkit detection problem as one of detecting invariant violations

• RQ2: How can we fetch memory pages without involving the target’s OS?

– Concerns our mechanism

– Answer: Leverage hardware advances

Research questions

Example 1: Linux Adore rootkit

int main()

open(…)

return(0)

sys_open(...)

sys_open

System call table

OS kernelUser app

int main()

open(…)

return(0)

sys_open(...)

evil_open(...)

malicious();

sys_open(...)

evil_open

System call table

OS kernelUser app

int main()

open(…)

return(0)

sys_open(...)

evil_open(...)

malicious();

sys_open(...)

evil_open

System call table

OS kernelUser app

Violated: Function pointer values in

system call table should not change

Example 2: Windows Fu rootkit

run_list

next_task

run_list

next_task

run_list

next_task

all-tasks: Used for process accounting

run-list: Used by the scheduler to select processes for execution

Process A Process B Process C

35Vinod Ganapathy - Policies and Mechanisms for OS Security

run_list

next_task

run_list

next_task

run_list

next_task

run_list

next_task

Hidden processProcess A Process B Process C

run_list

next_task

run_list

next_task

run_list

next_task

run_list

next_task

Hidden processProcess A Process B Process C

Violated: run-list ⊆ all-tasks

Urandom

Entropy Pool

(128 bytes)

Secondary

Entropy Pool

(128 bytes)Primary

Entropy Pool

(512 bytes)

/dev/random

/dev/urandom

External Entropy

Sources

Example 3: Kernel PRNG corruptor

Urandom

Entropy Pool

(128 bytes)

Secondary

Entropy Pool

(128 bytes)Primary

Entropy Pool

(512 bytes)

/dev/random

/dev/urandom

External Entropy

Sources

Example 3: Kernel PRNG corruptor

Attack: Modify coefficients of polynomials used to stir the entropy pools. Weaken quality of random numbers

Urandom

Entropy Pool

(128 bytes)

Secondary

Entropy Pool

(128 bytes)Primary

Entropy Pool

(512 bytes)

/dev/random

/dev/urandom

External Entropy

Sources

Example 3: Kernel PRNG corruptorViolated:

• poolinfo.tap1 ∈ {26, 103}

• poolinfo.tap5 == 1

Key technical challenges

• Vast attack surface:

– The kernel has thousands of data structures

• Specifying correctness properties:

– Infeasible to supply properties manually

Key technical challenges

• Vast attack surface:

– The kernel has thousands of data structures

– Solution: Use memory snapshots to analyze all

kernel data structures

• Specifying correctness properties:

– Infeasible to supply properties manually

– Solution: Infer invariants by adapting methods

from dynamic program analysis

Offline training phase

User app

Physical Memory

Hardware

Operating

System

SyscallProcess

Kernel

User app

Utilities &

Libraries

Clean reference machine Not rootkit infected

Analysis machine

Snapshot of

memory pages

Offline training phase

User app

Physical Memory

Hardware

Operating

System

SyscallProcess

Kernel

User app

Utilities &

Libraries

Clean reference machine Not rootkit infected

Analysis machine

Snapshot of

memory pages

Inference

Invariant DB

Online enforcement phase

User app

Physical Memory

Hardware

Operating

System

SyscallProcess

Kernel

User app

Utilities &

Libraries

Target machine Potentially rootkit infected

Analysis machine

Snapshot of

memory pages

Inference

Invariant DB

Compare

Prior work on inferring invariants

• Daikon: Dynamic program analysis tool to

infer data invariants [Ernst et al., 2000]

T1, T2, … , Tn = Traces from execution of a target

program, recording variable values

Values/properties invariant in in T1, T2, … , Tn

(e.g., foo == 5, foo ≤ bar + baz)

Adapting to memory snapshots

S1, S2, … Sn = Snapshots from reference machine

for (i ∈ [1 .. n]) {

Di = Reconstruct kernel data structures in Si

D1, D2, … , Dn

Data Structure Invariants

• sys_open == 0x3ee210fb

• run-list ⊆ all-tasks

• …

Reconstructing data structures

(3) Snapshot of

memory pages

(2) Entry-points into the kernel(1) Kernel data structure

type definitionsffffe400 init_task

ffffe410 phys_base

ffffe420 loops_per_jiffy

struct task_struct {...}

struct list_head {...}

struct siginfo {...}

(3) Snapshot of

memory pages

ffffe410 phys_base

struct task_struct {

int state;

int counter;

struct task_struct *next;

00000001

0034ea23

ac3456bc

Data at 0xffffe400Definition of task_struct

init_task.state = 1init_task.counter = 0x34ea23

init_task.next = 0xac3456bc

(3) Snapshot of

memory pages

ffffe410 phys_base

struct task_struct {

int state;

int counter;

struct task_struct *next;

Data at 0xac3456bcDefinition of task_struct

init_task.next.state = 0init_task.next.counter = 0x56ae71

init_task.next.next = 0xbf6723ae

00000000

0056ae71

bf6723ae

Experimental evaluation

① How effective is our approach at detecting

rootkits? i.e., what is the false negative rate?

②What is the quality of automatically-

generated invariants? i.e., what is the false

positive rate?

Target machine ran Linux 2.4.20. Used same

machine as reference machine as well.

Training phase

• Ran LMBench on reference machine:

– Collected 15 complete memory snapshots

(including reboots): took 25 minutes

– Inferred invariants using Daikon in 31 minutes

• Inferred 236,444 invariants across the

memory snapshots

False negative evaluation

• Conducted experiments with 23 Linux

rootkits:

– 14 rootkits from PacketStorm

– 9 advanced rootkits, discussed in the literature

• Installed rootkits one at a time on the target

machine and tested effectiveness of our

approach at detecting the infection

54November 30, 2009

Rootkit name Data structures affected Detected?

1. Adore-0.42 System call table (from PacketStorm)

2. All-root System call table (from PacketStorm)

3. Kbd System call table (from PacketStorm)

4. Kis-0.9 System call table (from PacketStorm)

5. Linspy2 System call table (from PacketStorm)

6. Modhide System call table (from PacketStorm)

7. Phide System call table (from PacketStorm)

8. Rial System call table (from PacketStorm)

9. Rkit-1.01 System call table (from PacketStorm)

10. Shtroj2 System call table (from PacketStorm)

11. Synapsys-0.4 System call table (from PacketStorm)

12. THC Backdoor System call table (from PacketStorm)

13. Adore-ng VFS hooks/UDP recvmsg (from PacketStorm)

14. Knark-2.4.3 System call table, proc hooks (from PacketStorm)

15. Disable Firewall Netfilter hooks (Baliga et al., 2007)

16. Disable PRNG VFS hooks (Baliga et al., 2007)

17. Altering RTC VFS hooks (Baliga et al., 2007)

18. Defeat signature scans VFS hooks (Baliga et al., 2007)

19. Entropy pool struct poolinfo (Baliga et al., 2007)

20. Hidden process Process lists (Petroni et al., 2006)

21. Linux Binfmt Shellcode.com

22. Resource waste struct zone_struct (Baliga et al., 2007)

23. Intrinsic DOS int max_threads (Baliga et al., 2007)

False positive evaluation

• Ran a benign workload for 42 minutes

– Copying Linux kernel source code

– Editing a text document

– Compiling the Linux kernel

– Downloading eight videos from Internet

– Perform file system operations using the IOZone

benchmark

• Only 82 out of 236,444 invariants spuriously

violated during execution

– Can be improved with more training

Current status of this approach

• Adopted widely in community for memory

snapshot-based rootkit detection

• Has led to numerous follow-on projects by

other research groups (200+ citations)

– More accurate data structure reconstruction

– Better ways to express invariants

– Improving accuracy of inferred invariants

Research questions

Tamper

resistance

Performance

isolation

Snapshot

consistency

Snapshot acquisition mechanism

Tamper

resistance

Performance

isolation

Snapshot

consistency

Tamper resistance

Target should not interfere with snapshot acquisition

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Tamper resistance

Physical Memory

Virtual Hardware

Operating

System

Hypervisor

• Hypervisor can fetch memory

from virtual machine without

OS involvement

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Co-processor

Tamper resistance

Physical Memory

Hardware

Operating

System• Co-processor uses DMA

• OS on target involved in

DMA setup

• Malicious OS can hide

portions of memory with

malicious content

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Co-processor

Performance isolation

Do not halt the target during snapshot acquisition

• Necessary for situations where continuous

snapshot acquisition is necessary

• Hypervisor-based acquisition requires pausing

the virtual machine

• Co-processor can operate in concert with target

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Co-processor

Snapshot consistency

Snapshot should faithfully represent target’s state at a given instant in time

Physical Memory

Hardware

Operating

System F1 F2…T

F1 F2…T + δ

CONSISTENT

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Co-processor

Snapshot consistency

Snapshot should faithfully represent target’s state at a given instant in time

Physical Memory

Hardware

Operating

System F1 F2…

T T + δ

Co-processor cannot pause target.

Snapshot may contain pages

obtained at different instants in time

INCONSISTENT

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Co-processor

3D-stacking

Our contribution

• Based on 3D-stacked technology:

– New hardware manufacturing technology that

“stacks” memory/processing logic atop the chip

– Early versions of 3D-stacked hardware already

on market, e.g., AMD Radeon series

3D-stacked chip

Picture courtesy of AMD

CPU and

Memory controller

On-chip memory

(high-speed)

3D-stacked chip

Picture courtesy of AMD

CPU and

Memory controller

On-chip memory

(high-speed)

Memory bus

Traditional (off-chip)

DRAM memory

Our use of 3D-stacking

• On-chip DRAM treated as a page-granularity

cache of off-chip DRAM memory

– Every address accessed by the CPU will result in

the page frame being fetched to on-chip DRAM

Cache of off-chip

DRAM memoryOff-chip DRAM

On-chip DRAM

Memory controller

Crypto logic

Memory bus

Triggering snapshot acquisition

On-chip DRAM

Memory controller

Crypto logic

Trigger = Device that communicates to the

CPU to enter snapshot acquisition mode: Physical device attached to South/NorthBridge

that sends a non-maskable interrupt

NIC with Wake-on-LAN-like feature

Off-chip DRAM

Memory bus

Snapshot acquisition mode

Memory controller

Crypto logic

• Memory controller splits on-chip DRAM into

two parts: Cache of off-chip DRAM memory

Copy-on-Write (CoW) area

CacheCoW

Off-chip DRAM

Memory bus

Memory controller

Crypto logic

• Hardware brings one page frame of off-chip

DRAM at a time to on-chip DRAM cache

CacheCoWFiFi

Off-chip DRAM

Memory bus

Memory controller

Crypto logic

• Crypto logic digitally signs contents of page: Random nonce used to prevent replay attacks

Same nonce used for all pages in snapshot

CacheCoW

+ Page#

+ Rand#Fi

Off-chip DRAM

Memory bus

Memory controller

Crypto logic

• Hardware instructs OS to write signed page

to external medium: Even if OS is infected, it cannot cheat, since

integrity of page is protected by the hardware

CacheCoW

+ Page#

+ Rand#Fi

Off-chip DRAM

Memory bus

• CPU continues to execute concurrently: If it writes to page Fj that has not yet been

copied Memory controller makes a copy of the

original page in the Copy-on-Write area

When hardware ready to snapshot Fj, copy

created from Copy-on-Write area

Memory controller

Crypto logic

Memory bus

CacheCoW

Off-chip DRAM

At conclusion of acquisition

• Consistent snapshot of off-chip memory at

instant when acquisition was initiated

• Snapshot is tamper-resistant even to a

corrupted OS

• Obtained without pausing target machine

Security analysis

Malicious OS cannot:

Corrupt pages in snapshot: Integrity

Hide pages from snapshot: Completeness

Replay old snapshot: Freshness

“Clean” itself during snapshot acquisition

because Copy-on-Write stores original page:

External control

Evaluation

• Atop 3D-stacked hardware emulator

• Evaluated:

– Impact of 3D-stacked memory available

– Effectiveness of performance-isolation claim

• Used canneal, memcached, graph500, mcf

– Time to procure full snapshot of memory:

• ~10-100 seconds, depending on external medium

– Complexity of hardware modifications:

• Evaluated using CACTI and Aladdin

• Negligible area/energy overheads

Evidence of performance isolation

Applications make progress as long as space available

in on-chip CoW area. Space in CoW area dependent on

speed of external medium that stores snapshot

Other research projects…

• Improving cloud platform security [ACSAC’08, RAID’10, CCS’12a, SOCC’14]

• Security for mobile devices (and other IoT devices)[MobiSys’11, TIFS’13, MobiSys’16]

• Hardware support for software and system security [CCS’08, ECOOP’12a, TIFS’13, MobiSys’16, RU-DCS-TR724]

• Web application and Web browser security [ACSAC’08, ACSAC’09, ECOOP’12a, ECOOP’12b, ECOOP’14, FSE’14]

• Tools for cross-platform mobile app development [ICSE’13, ASE’15]

• Retrofitting legacy software for security [CCS’05, Oakland’06, ASPLOS’06, ICSE’07, CCS’08, CCS’12b]

• Proofs of security for retrofitting transformations[Work in progress]

Generic theme: Computer Systems Security

A big thank you to my students

Graduated PhDs

• Dr. Mohan Dhawan (IBM Research India)

• Dr. Saman Zarandioon (Amazon.com)

• Dr. Shakeel Butt (NVidia now at Google)

• Dr. Liu Yang (HP Labs now at Baidu)

• Dr. Rezwana Karim (Samsung Research America)

• Dr. Amruta Gokhale (Teradata)

Former Postdocs

• Dr. Arati Baliga (AT&T Security Labs)

Graduated MS students

• Jeffrey Bickford (AT&T Research)

• Yogesh Padmanaban (Microsoft)

Current PhD students

• Jay P. Lim, Hai Nguyen, Daeyoung Kim.

URL: http://www.cs.rutgers.edu/~vinodg

Email: vinodg@cs.rutgers.edu

Policies and Mechanisms for Operating System Securityvg/talks/os-integrity/osi.pdf · Example 2:...

Documents

Rootkit on Linux X86 v2.6

A Comparitive Analysis of Rootkit Detection Techniquessce.uhcl.edu/yang/research/A Comparitive Analysis of Rootkit... · A COMPARATIVE ANALYSIS OF ROOTKIT DETECTION TECHNIQUES by

2012-08-14 Experimenting with live Cyberattackers for ... · rootkit test rootkit test rootkit test rootkit test rootkit 26 close ftp 27 . 29 30 . decompress decompress secureport

Trojan, Backdoors,RootKit

TDL3 Rootkit Background

Rootkit for the Masses

ROOTKIT -MALWARE

The Rootkit Arsenal

Rootkit and Botnet

Aula 2 - Modelo OSI.pdf

Uroburos: the snake rootkit

شرح Layers Osi.pdf

Rootkit 101 - 2nd Edition

3. windows system과 rootkit

Rootkit presentation

Cybercrime und Cyberwar - Linuxwochen · 2015. 2. 6. · Virtualization Rootkit Blue Pill Bootloader Rootkit / Bootkit Evil Maid Attack Hardware/firmware Rootkit 07.05.2013 52 Botnetze

Yet Another Android Rootkit

Các chuẩn mạng và mô hình OSI.pdf

Rootkit List

r77 Rootkit - bytecode77.com