Subroutine based detection of APT malware

Joseph Sexton, Curtis Storlie, Blake Anderson

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Statistical detection of mass malware has been shown to be highly successful. However, this type of malware is less interesting to cyber security officers of larger organizations, who are more concerned with detecting malware indicative of a targeted attack. Here we investigate the potential of statistically based approaches to detect such malware using a malware family associated with a large number of targeted network intrusions. Our approach is complementary to the bulk of statistical based malware classifiers, which are typically based on measures of overall similarity between executable files. One problem with this approach is that a malicious executable that shares some, but limited, functionality with known malware is likely to be misclassified as benign. Here a new approach to malware classification is introduced that classifies programs based on their similarity with known malware subroutines. It is illustrated that malware and benign programs can share a substantial amount of code, implying that classification should be based on malicious subroutines that occur infrequently, or not at all in benign programs. Various approaches to accomplishing this task are investigated, and a particularly simple approach appears the most effective. This approach simply computes the fraction of subroutines of a program that are similar to malware subroutines whose likes have not been found in a larger benign set. If this fraction exceeds around 1.5 %, the corresponding program can be classified as malicious at a 1 in 1000 false alarm rate. It is further shown that combining a local and overall similarity based approach can lead to considerably better prediction due to the relatively low correlation of their predictions.

Original languageEnglish (US)
Pages (from-to)1-9
Number of pages9
JournalJournal of Computer Virology and Hacking Techniques
DOIs
StateAccepted/In press - Dec 21 2015
Externally publishedYes

Fingerprint

Subroutines
Malware
Classifiers

Keywords

  • APT
  • Malware detection
  • Static analysis
  • Subroutine similarity

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Hardware and Architecture
  • Computational Theory and Mathematics
  • Software

Cite this

Subroutine based detection of APT malware. / Sexton, Joseph; Storlie, Curtis; Anderson, Blake.

In: Journal of Computer Virology and Hacking Techniques, 21.12.2015, p. 1-9.

Research output: Contribution to journalArticle

@article{2fc777272336427896d0f10ef91835d7,
title = "Subroutine based detection of APT malware",
abstract = "Statistical detection of mass malware has been shown to be highly successful. However, this type of malware is less interesting to cyber security officers of larger organizations, who are more concerned with detecting malware indicative of a targeted attack. Here we investigate the potential of statistically based approaches to detect such malware using a malware family associated with a large number of targeted network intrusions. Our approach is complementary to the bulk of statistical based malware classifiers, which are typically based on measures of overall similarity between executable files. One problem with this approach is that a malicious executable that shares some, but limited, functionality with known malware is likely to be misclassified as benign. Here a new approach to malware classification is introduced that classifies programs based on their similarity with known malware subroutines. It is illustrated that malware and benign programs can share a substantial amount of code, implying that classification should be based on malicious subroutines that occur infrequently, or not at all in benign programs. Various approaches to accomplishing this task are investigated, and a particularly simple approach appears the most effective. This approach simply computes the fraction of subroutines of a program that are similar to malware subroutines whose likes have not been found in a larger benign set. If this fraction exceeds around 1.5 {\%}, the corresponding program can be classified as malicious at a 1 in 1000 false alarm rate. It is further shown that combining a local and overall similarity based approach can lead to considerably better prediction due to the relatively low correlation of their predictions.",
keywords = "APT, Malware detection, Static analysis, Subroutine similarity",
author = "Joseph Sexton and Curtis Storlie and Blake Anderson",
year = "2015",
month = "12",
day = "21",
doi = "10.1007/s11416-015-0258-7",
language = "English (US)",
pages = "1--9",
journal = "Journal of Computer Virology and Hacking Techniques",
issn = "2274-2042",
publisher = "Springer Science + Business Media",

}

TY - JOUR

T1 - Subroutine based detection of APT malware

AU - Sexton, Joseph

AU - Storlie, Curtis

AU - Anderson, Blake

PY - 2015/12/21

Y1 - 2015/12/21

N2 - Statistical detection of mass malware has been shown to be highly successful. However, this type of malware is less interesting to cyber security officers of larger organizations, who are more concerned with detecting malware indicative of a targeted attack. Here we investigate the potential of statistically based approaches to detect such malware using a malware family associated with a large number of targeted network intrusions. Our approach is complementary to the bulk of statistical based malware classifiers, which are typically based on measures of overall similarity between executable files. One problem with this approach is that a malicious executable that shares some, but limited, functionality with known malware is likely to be misclassified as benign. Here a new approach to malware classification is introduced that classifies programs based on their similarity with known malware subroutines. It is illustrated that malware and benign programs can share a substantial amount of code, implying that classification should be based on malicious subroutines that occur infrequently, or not at all in benign programs. Various approaches to accomplishing this task are investigated, and a particularly simple approach appears the most effective. This approach simply computes the fraction of subroutines of a program that are similar to malware subroutines whose likes have not been found in a larger benign set. If this fraction exceeds around 1.5 %, the corresponding program can be classified as malicious at a 1 in 1000 false alarm rate. It is further shown that combining a local and overall similarity based approach can lead to considerably better prediction due to the relatively low correlation of their predictions.

AB - Statistical detection of mass malware has been shown to be highly successful. However, this type of malware is less interesting to cyber security officers of larger organizations, who are more concerned with detecting malware indicative of a targeted attack. Here we investigate the potential of statistically based approaches to detect such malware using a malware family associated with a large number of targeted network intrusions. Our approach is complementary to the bulk of statistical based malware classifiers, which are typically based on measures of overall similarity between executable files. One problem with this approach is that a malicious executable that shares some, but limited, functionality with known malware is likely to be misclassified as benign. Here a new approach to malware classification is introduced that classifies programs based on their similarity with known malware subroutines. It is illustrated that malware and benign programs can share a substantial amount of code, implying that classification should be based on malicious subroutines that occur infrequently, or not at all in benign programs. Various approaches to accomplishing this task are investigated, and a particularly simple approach appears the most effective. This approach simply computes the fraction of subroutines of a program that are similar to malware subroutines whose likes have not been found in a larger benign set. If this fraction exceeds around 1.5 %, the corresponding program can be classified as malicious at a 1 in 1000 false alarm rate. It is further shown that combining a local and overall similarity based approach can lead to considerably better prediction due to the relatively low correlation of their predictions.

KW - APT

KW - Malware detection

KW - Static analysis

KW - Subroutine similarity

UR - http://www.scopus.com/inward/record.url?scp=84950236042&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84950236042&partnerID=8YFLogxK

U2 - 10.1007/s11416-015-0258-7

DO - 10.1007/s11416-015-0258-7

M3 - Article

AN - SCOPUS:84950236042

SP - 1

EP - 9

JO - Journal of Computer Virology and Hacking Techniques

JF - Journal of Computer Virology and Hacking Techniques

SN - 2274-2042

ER -