Embed Size (px)
Citation preview
1
Malware Detection
Prof. Ravi SandhuExecutive Director and Endowed Chair
Lecture 14
© Ravi Sandhu World-Leading Research with Real-World Impact!
CS 5323
Virus detection is undecidable Cohen dissertation (1985), paper (1987)
Anti-virus (more generally anti-malware) is a great business model Need regular updates Infinite supply of new malware
Malware can be stealthy Malware can be really stealthy
© Ravi Sandhu 2World-Leading Research with Real-World Impact!
Highlights
© Ravi Sandhu 3World-Leading Research with Real-World Impact!
Malware Detection Techniques
Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007.
© Ravi Sandhu 4World-Leading Research with Real-World Impact!
Malware Detection Techniques
Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007.
MisuseDetection
Behavior-BasedDetection
© Ravi Sandhu 5World-Leading Research with Real-World Impact!
Signature Limitations
Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007.
S needsregularupdates
© Ravi Sandhu 6World-Leading Research with Real-World Impact!
Anomaly Based
TrainingPhase
DetectionPhase
Inferpatterns
Inferspecifications
© Ravi Sandhu 7World-Leading Research with Real-World Impact!
Anomaly Based Limitations
Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007.
Blue area is false positivesIf white area extends outside blue area we have false negatives
Defeat signature-based detection Encrypted malware Polymorphic malware Metamorphic malware
Rootkit can misrepresent the existence or content of executable files
© Ravi Sandhu 8World-Leading Research with Real-World Impact!
Stealthy Malware
You, I., and Yim, K. Malware obfuscation techniques: A brief survey. IEEE International Conference on Broadband, Wireless Computing, Communication and Applications, Nov 2010, pp. 297-300.
© Ravi Sandhu 9World-Leading Research with Real-World Impact!
Encrypted Malware
Encrypted Main Body Decryptor Cleartext Main Body
Encrypted Main Body Key’ Decryptor
execute malware
Key
propagatemalware
© Ravi Sandhu 10World-Leading Research with Real-World Impact!
Encrypted Malware
Encrypted Main Body Decryptor Cleartext Main Body
Encrypted Main Body Key’ Decryptor
execute malware
Key
propagatemalware
revealssignature
© Ravi Sandhu 11World-Leading Research with Real-World Impact!
Polymorphic Malware
Encrypted Main Body Decryptor Cleartext Main Body
Encrypted Main Body Key’ Obfuscated Decryptor
execute malware
Key
propagatemalware
© Ravi Sandhu 12World-Leading Research with Real-World Impact!
Polymorphic Malware
Encrypted Main Body Decryptor Cleartext Main Body
Encrypted Main Body Key’ Obfuscated Decryptor
execute malware
Key
propagatemalware
nosignature
© Ravi Sandhu 13World-Leading Research with Real-World Impact!
Polymorphic Malware
Encrypted Main Body Decryptor Cleartext Main Body
Encrypted Main Body Key’ Obfuscated Decryptor
execute malware
Key
propagatemalware
nosignature
Execute in a sandbox and detect the signature after
decryption
© Ravi Sandhu 14World-Leading Research with Real-World Impact!
Polymorphic Malware
Encrypted Main Body Decryptor Cleartext Main Body
Encrypted Main Body Key’ Obfuscated Decryptor
execute malware
Key
propagatemalware
nosignature
Execute in a sandbox and detect the signature after
decryptionMutation Engines automate this construction
© Ravi Sandhu 15World-Leading Research with Real-World Impact!
Metamorphic Malware
Original Main Body Original Main Body
execute malware
propagatemalware
nosignature
Obfuscated Main Body
execute malware
Obfuscated Main Body
Obfuscated Main Body
execute malware
Obfuscated Main Body
propagatemalware
Dead-Code Insertion Register Reassignment Subroutine Reordering Instruction substitution Code transposition Code Integration
© Ravi Sandhu 16World-Leading Research with Real-World Impact!
Obfuscation Techniques
Not visible in source code Reappears in binary code due to malware infected
compiler In theory could reappear in binary code due to other
components in binary execution workflow Loader Linker OS BIOS
© Ravi Sandhu 17World-Leading Research with Real-World Impact!
Really Stealthy Malware
Ken Thompson. Reflections on trusting trust. Commun. ACM 27, 8 (August 1984), 761-763.
© Ravi Sandhu 18World-Leading Research with Real-World Impact!
Malicious Compiler Inserts a Backdoor
Malicious CompilerBinary
OS Login module
Infected Login Binary
© Ravi Sandhu 19World-Leading Research with Real-World Impact!
Malicious Compiler Inserts a Backdoor
Malicious CompilerBinary
OS Login module
Infected Login Binary
Assumption: Malicious behavior cannot be detected
in binary, but may be detectable in
compiler source
© Ravi Sandhu 20World-Leading Research with Real-World Impact!
Self-Compiler
Compiler binary for language L
Compiler source for language L
Compiler binary for language L
© Ravi Sandhu 21World-Leading Research with Real-World Impact!
Malicious Self-Compiler in Binary and Source
Compiler binary for language L
Malicious Compiler source for language L
Malicious Compiler binary for language L
© Ravi Sandhu 22World-Leading Research with Real-World Impact!
Malicious Self-Compiler in Binary and Source
Compiler binary for language L
Malicious Compiler source for language L
Malicious Compiler binary for language L
Source code analysis will reveal malicious behavior
© Ravi Sandhu 23World-Leading Research with Real-World Impact!
Doubly Malicious Self-Compiler in Binary and Source
Compiler binary for language L
Doubly Malicious Compiler source for language L
Doubly Malicious Compiler binary for language L
Source code analysis will reveal doubly malicious
behavior
© Ravi Sandhu 24World-Leading Research with Real-World Impact!
Doubly Malicious Complier Binary Behavior
Doubly Malicious Compiler binary for language L
Compiler source for language L
Doubly Malicious Compiler binary for language L
OS Login module
Infected Login Binary
Doubly Malicious Compiler binary for language L
© Ravi Sandhu 25World-Leading Research with Real-World Impact!
Doubly Malicious Complier Binary Behavior
Doubly Malicious Compiler binary for language L
Compiler source for language L
Doubly Malicious Compiler binary for language L
OS Login module
Infected Login Binary
Doubly Malicious Compiler binary for language L
No trace of malicious behavior
in source code
© Ravi Sandhu 26World-Leading Research with Real-World Impact!
Malicious Self-Compiler in Binary but not in Source
Malicious Compiler binary for language L
Compiler source for language L
Malicious Compiler binary for language L
Wheeler, D.A., Countering trusting trust through diverse double-compiling, 21st Annual Computer Security Applications Conference, pp.13-48, 5-9 Dec. 2005.
partial countermeasure
![Whitehead's Problem is Undecidable - Matematikai Intézetweb.cs.elte.hu/~agoston/bboard/fg12tav/Eklof.pdf · 1976] WHITEHEAD'S PROBLEM IS UNDECIDABLE 777 homomorphism p: A -> B such](https://img.pdfslide.net/doc/110x75/5a7073e27f8b9abb538bf309/whiteheads-problem-is-undecidable-matematikai-intzetwebcseltehuagostonbboardfg12taveklofpdfpdf.jpg)
