Operating System Under Attack - Amazon Web Servicesecwp.s3.amazonaws.com/.../uploads/sites/32/2015/02/Operating-System-Under-Attack.pdfoperating system. We will take a look under the

Operating System Under Attack

Operating System Under Attack

James L. Antonakos

SUNY Distinguished Teaching Professor

Computer Science, Broome Community College

NCI Fellow

Operating System Fundamentals

Let us look at the block diagram of a typical modern

operating system.

We will take a look under the hood and see what is going

on inside the operating system.

This knowledge is essential for the forensic examiner, as

he or she must understand the inner workings of the

operating system, its structures and activities, in order to

know where to look for information and evidence.

Analyzing malware requires essentially the same

knowledge, as the malware will take advantage of

existing operating system code and data structures for

its own purposes.

2


3


Process Management: This component is responsible

for scheduling new processes and switching between

processes that are currently running.

In older operating systems, only one process could

execute at a time. With the technique of task switching,

multiple processes are able to execute seemingly

simultaneously on a single microprocessor by giving

each process a slice of time to execute, then suspending

the process, saving all processor registers, and then

loading all the registers with data from a previously-

saved process and resuming the process where it left

off.

4


A process queue is used to maintain the set of all

running processes and switch between them.

Some processes are given a higher priority than others

and thus receive more slices of time to execute than

others.

Since two or more processes may also need to

communicate with each other, the Process Management

section provides the mechanism for interprocess

communication, either through a shared memory area or

through messages.

5


Memory Management: This component is responsible

for managing the RAM in the computer system.

RAM is divided into fixed-sized pages (for example, 4 KB

chunks) and allocated on a demand basis, hence the

name demand paging.

When a new process begins execution, it is allocated a

set of RAM pages. It is important to note here that the

entire process does not have to exist in memory for it to

begin execution. If the process attempts to execute an

instruction from a page that has not yet been loaded into

memory, a page fault will occur and the required page

will be loaded into memory from the hard disk.

6


If all pages have been allocated, then an existing page

must be chosen for replacement when a page fault

occurs.

The victim page may be simply overwritten with the new

page.

If the victim page has been modified, however, it must be

first copied back to the hard disk.

When there is a large number of processes in memory,

the amount of page swapping between the hard disk and

memory can become excessive, a condition called

thrashing.

7


In this case, the performance of the operating system

suffers as too much time is being spent moving pages

back and forth between RAM and the hard disk, and not

enough time spent executing instructions.

Terminating processes, or making more RAM available,

will eliminate the thrashing.

The Memory Management component also periodically

recovers pages that are no longer needed but for one

reason or another have not been returned to the free

storage pool of pages. This activity is called garbage

collection.

8


I/O Management: When you are typing on the computer

keyboard, what is happening inside the computer?

Every keystroke generates an interrupt, an electrical

signal that causes the microprocessor to break away

from what it is doing and execute an interrupt service

routine to handle the interrupt.

In the case of the keyboard, its interrupt service routine

places the key code into a memory buffer.

When the buffer gets filled, it will accept no more keys

until those already in the buffer are processed.

9


Sometimes the system gets busy with a particular chore,

such as reading data from the hard disk, and cannot

devote time to processing keystrokes, and the buffer fills

up.

This is just one example of why even the I/O operations

in a computer system need to be managed.

Since the actual hardware employed in a computer

system may vary widely, special code modules called

drivers are used to translate generic operating system

operations, such as “read a network packet” or “draw a

window on the video display” into the actual instructions

necessary to control the associated I/O device.

10


A portion of the I/O Management component also

manages the file system or file systems used by the

operating system.

The file system is designed to organize files and folders

and make it easier for users to create, edit, and delete

files.

The file system understands how to access and utilize

many different kinds of storage devices, from hard disks,

to USB thumb drives, CDs, and DVDs.

11


User Management: This component is responsible for

authenticating users during the logon process and

maintaining settings and permissions.

These settings and permissions are unique to each user.

For example, in a Windows system, the user gets his or

her own customized Desktop after logging into the

system, filled with their own shortcuts and configured to

their liking.

A user’s permissions dictate what the user is or is not

allowed to do on the system, such as access certain files

or folders, install software, or make changes to the

system settings, including their own permissions.

12


Resource Management: This component manages all

devices that may be scheduled for use, such as the

processor(s), printer, hard disk, and other I/O devices.

One of the responsibilities for the Resource

Management component is detecting and avoiding

deadlock conditions within the system.

For example, imagine that one user has been assigned

the printer and then requests the hard disk. Another user

has been assigned the hard disk and requests the

printer. Since neither user can go forward until both

resources are available, they become deadlocked, and

neither is allowed to proceed.

13


Tying everything together and maintaining control over

all areas of the operating system is the System

Executive, or Kernel.

The System Executive establishes cooperation between

the different management components, among other

things.

For example, when a new window must be opened on

the video display, the I/O management component

requests a block of memory from the Memory

management component to provide storage for the

portion of the display that will be covered by the window.

14


The code contained within the System Executive

executes at a higher level of privilege than user

processes and even other system processes.

This privilege level is provided in part by the underlying

microprocessor the code executes on.

In the case of the 80x86, this is due to its Protected

Mode of operation.

Other microprocessors provide User and Supervisor

modes of execution.

These modes of execution provide access to privileged

instructions and operations not allowed otherwise.

15


This has the benefit of enabling the System Executive to

control how memory is accessed and used.

With its power and centralized control over the entire

system, it is no wonder that the System Executive is of

great interest to malware coders, some of which want

nothing less than total control over the operating system.

By targeting areas of the System Executive, a malware

coder will be able to tap into the protected, inner

workings of the operating system.

An example of this is a key logger, a program designed

to capture and store every keystroke entered on the

keyboard.

16


The keystroke storage may be periodically transmitted

over the Internet to another machine for analysis to

locate usernames, passwords, and other valuable and

sensitive information.

The key logger may work by hooking the keyboard

interrupt service routine.

A hooked interrupt service routine will first make a call to

another routine, one provided by the malware for illicit

purposes.

When this routine finishes, it returns back to the original

interrupt service routine to complete the normal

processing.

17


One way to prevent malware from hooking an interrupt

service routine is to randomize the address of the routine

at boot time.

Malicious code will often take advantage of known

operating system details in order to exploit them, such as

the addresses of interrupt service routines that remain

unchanged within a particular version of the operating

system.

By randomizing the addresses of the interrupt service

routines, the malware does not have a guaranteed entry

point into the System Executive.

18


This is one of the features added to Windows Vista to

help protect the operating system from attacks.

The System Executive may also be compromised by a

rootkit, the most sophisticated malware used to take

over a system.

A rootkit replaces critical operating system files that

operate at the kernel level with its own malicious

versions of them.

These malicious files provide the functionality of the

original files, plus additional malicious functionality.

A rootkit is very difficult to remove.

19

Vulnerabilities

There are many vulnerabilities in the operating system

that malware is able to exploit. These include:

Operating system programs (.EXE and .DLL files)

Driver files

Application programs

Incorrectly configured security settings

User programs

User accounts (Administrator privileges = trouble waiting to

happen)

The user !!!

20

Attack Vectors

An attack vector is a method by which malware is

introduced into the operating system. These include:

Infected email attachments (executable files, documents

containing malware [infected PDFs] or malicious macros [MS

Office document with Visual BASIC script]

Infected web pages (drive-by downloads)

Trojan horse programs or other programs infected with malware

that are downloaded intentionally

Infected USB Thumb drive [Is it still ok to mention floppy disks?]

Network messages (SQL Slammer)

Buffer overflows to exploit all sorts of applications

RPC (Remote Procedure Calls) via network to vulnerable

services

21

Attack Vectors

A successful attack

does not have to

contain a large

amount of code.

Consider the SQL

Slammer worm sent

using the UDP

protocol… it only

contained 376 bytes !

22

Attack Vectors

Why is it so hard to keep malware from entering a system?

Zero-day attacks: This malware is so new, none of the AntiVirus

companies know about it or have signatures their scanning

software can recognize.

Obfuscated script code: Many web pages, or files attached to

emails, or even files containing script code, contain javascript or

VisualBASIC script. Often there is normal, non-malicious purpose

for these scripts. But malicious scripts are obfuscated, their

functionality disguised, in order to evade detection.

23

Attack Vectors

No AntiVirus scanner or IDS is

going to parse through the

obfuscated code to see if it is

malicious.

This is a job for a human

investigator, or the code

needs to be executed in a

sandbox to see what it does.

A first pass through the code

shows a little more

organization, but still no clue

what is going on.

24

Attack Vectors

But when the script is fully decoded it reveals its

downloading capability (only a portion of the code is shown

here):

25

Attack Vectors

Since the malware writers know that their malicious code

files will be scanned of they reside on disk, they have

evolved new ways of infecting a system without having to

actually store the code in a file.

One technique involves code injection, where the

malicious code is injected (written) into the memory space

of a running process, and lives as long as that process

lives.

Another technique involves writing a special key into the

Registry that contains the malicious code, so when the

Registry is processed at boot time, the code is loaded into

RAM.

26

Other Threats

There are other threats to an operating system that do

not come from malware. These include:

Unauthorized access. The user walked away from their

computer and did not lock the Desktop.

Weak password. An attacker uses a network-based program to

brute-force crack a user’s password. Or they use a bootable CD

or USB Thumb drive running Ophcrack to decrypt passwords.

Improper permissions on user account. A hacker using a

different system is able to compromise the user’s computer by

installing software on shared drives or making other changes to

the system.

Non-protection of data. It is a fact that laptops get stolen all the

time, with their hard drives full of sensitive data that is not

encrypted.

27

Other Threats

There are two ways to protect sensitive files from

unauthorized access:

Encrypt the files individually. A key is then needed to decrypt

them.

Encrypt the entire hard disk (using BitLocker or some other tool).

This prevents the disk from being read if the computer is

stolen…. but not if the system is already booted, since whole

disk encryption is transparent to the user… which is why you

want to also encrypt sensitive files individually as well.

Windows Firewall: Is it even on? By default, the firewall

will block unauthorized programs from accessing the

network, unless the user allows it and adds an

exception.

28

Other Threats

The vulnerable Registry:

The Windows Registry is the master database maintained by the

operating system that keeps track of all hardware and software

settings and controls the behavior of the system.

The Registry consists of keys that contain information about a

particular object.

Some keys that are useful to hackers control the following:

What programs and services start automatically when Windows boots.

The DNS settings used when looking up domain names.

System security settings that enable the malware to perform its activities.

System security settings that prevent the user from doing things, such as

opening Task manager to view running processes.

Create scheduled tasks that run malware at certain times.

29

Other Threats

Some common Registry key launch points:

30

Other Threats

What other ways does malware use to make itself

“persistent?”

In other words, how does malware survive a reboot?

Put a copy of itself, or a link to itself, inside the Startup folder.

Trojanize a system binary (infect a system file with a copy of

itself).

In order to evade detection, malware will also search for

any installed AntiVirus software and either stop it, or

make changes to the HOSTS file ( located at

C:\Windows\System32\drivers\etc ).

31

Other Threats

By making changes to the HOSTS file, malware can cause the

operating system to block access to AntiVirus websites and prevent

the software from downloading updates to their signature files.

Here’s a secret: access to malware sites can be blocked the same

way !

32

Other Threats

Not installing updates is a risky practice.

Updates are pushed out for a reason: they patch flaws in

the operating system or its application programs.

Why do so many of the older malware attacks still work

today? Because people do not patch their systems.

33

Conclusion

You need to have a good understanding of the

organization and functionality of an operating system to

really be able to protect it against threats.

The typical user may not have all the necessary

knowledge and skill needed to adequately protect their

computers, or investigate and clean an infection.

Security awareness training continues to be a critical tool

in educating users on how to use their computers safely.

34

Thank You

James Antonakos

SUNY Distinguished Teaching Professor

Computer Science, Broome Community College

NCI Fellow

[email protected]

35

mailto:[email protected]

IAPP CPE Credit

CPE Submission Form:

https://privacyassociation.org/certify/submit-cpe/




Documents

Operating System Under Attack - Amazon Web Servicesecwp.s3.amazonaws.com/.../uploads/sites/32/2015/02/Operating-System-Under-Attack.pdfoperating system. We will take a look under the