A Guide to Malware Detection Techniques: AV, NGAV, and Beyond
Back in the early days of malware, scammers spread their wares around from user to user via infected floppy disks. As computer usage became more common, it became easier for attackers to spread their malicious code. Understanding the risks of these unmitigated viruses, early software pioneers such as G Data and Sophos developed programs with the intent of identifying and blocking these threats, which were referred to as anti-virus programs. This method of malware prevention is now called signature-based detection.
(To learn more how Cynet can help you protect from all types of threats, click here)
Using signature-based detection, each file is analyzed, assigned a signature or hash (a unique alphanumeric way to identify malware), and then added to the signature database, where it’s used for comparison in subsequent malware incidents. When a suspicious file is found on a computer running the antivirus (AV) software, the program looks for patterns that may match a known malware family. If a match is made with a known variant, it’s blocked.
Today, due to constantly evolving malware, signature-based antivirus software is no longer an effective solution. In this article, we’ll look at the evolution of malware detection techniques, ranging from signature-based detection to next-generation antivirus (NGAV) software—and beyond.
Shortcomings of Signature-Based Detection
Signatures were the mainstay of malware detection techniques for years, and, in truth, they are still useful when it comes to known threats, such as older, still active ones. But when new unknown variants strike, which is far more common today, signatures are less effective.
Modern malware employs many techniques to evade detection, making it harder for signature-matching software to find it. Here’s a look at some forms of malware that do this.
Polymorphic Malware Variants
Polymorphic malware constantly changes its features to avoid being detected. Malware creators use polymorphism to bypass pattern-matching detection techniques used in antivirus programs. Using a mutation engine, polymorphic malware can change certain features, such as file names and/or hashes, which allows it to successfully hide its code. Today, 97% of malware uses polymorphic techniques, whose signatures change with every iteration. This means that while these variants are based on known malware families, they are undetectable by signature-based detection solutions.
Code obfuscation is another set of techniques used to evade detection and analysis. Obfuscation makes source code very difficult to understand, or even unreadable, in order to bypass static analysis tools. It uses multiple methods to hide code, including making instruction changes, inserting dead code to change the appearance of the malware, packing to compress malware, and crypting (which encrypts malware).
Abuse of Legitimate Signed Windows Binaries
Attackers may use already signed binaries or components such as Mavinject to execute malicious code inside processes. This allows the code to avoid signature-based antivirus methods. According to MITRE ATT&CK, using legitimate programs in suspicious ways is a good indication that you may be under attack.
Campaigns Using Malware Novelties
Almost all modern malware campaigns employ one or more detection evasion mechanisms. Here are some of the most notorious examples that could easily bypass signature-based detection.
Qbot initially debuted in 2009 as a trojan that stole financial information. While it’s been a threat to businesses and individuals since it surfaced, in 2019, it began displaying a new feature set to evade detection. In its enhanced form, it uses anti-analysis methods and obfuscation to slip past AV and AM. It also changes C2 servers often, and, according to Varonis, the security firm who first located the new and improved version of Qbot, it can create new files with seemingly random names.
Emotet is an advanced, polymorphic, modular banking trojan that can evade 75% of antivirus programs. It uses a few different methods to maintain persistence. When we say it’s modular, we mean that it uses different modules of itself to change how it affects systems. Emotet is also able to tell if it’s targeting a virtual machine, which prompts it to become inactive.
Stuxnet is perhaps the most famous zero-day vulnerability ever. It made headlines in 2010 when it took down a portion of the nuclear reactor at Iran’s Natanz uranium enrichment facility. What made Stuxnet unique was its ability to jump from the digital world to the physical world, where it actually destroyed a portion of the reactors.
Evolution of Malware Protection Techniques
It’s clear that signature-based detection alone cannot stand up to these evolving threats. So as malware has advanced, we’ve had to create solutions to keep up. Thus, antivirus software creators introduced new tools and methods to rout out malware, including:
Heuristic analysis looks at the behavior of a system or software to detect new threats that are not found by signature-based solutions. Initially, it establishes a baseline of normal activity for the system or software, and then, when something different occurs, it stands out as an anomaly.
Heuristic analysis is one of the only malware detection techniques that can detect polymorphic malware. What’s more, it allows developers to continuously change rules based on new threats, and it does not give details on how threats are flagged to malware developers. But, when code is sufficiently obfuscated, heuristic analysis will not be able to detect it.
Sandboxes detect malware by testing potentially malicious code in an isolated virtual environment. This allows researchers to observe the code’s real behavior in a safe environment, where it cannot spread or do any harm to the system and network it’s running on.
Sandboxing is a useful malware detection technique, as it is able to determine, in great detail, how files really act in environments. It can also provide in-depth information on behaviors, allowing analysts to figure out what the intentions of a given threat were.
However, the sandbox method has some significant drawbacks. Modern threat actors create “sandbox-aware” malware that knows when it’s being run in a sandbox, and, therefore, acts differently than it would in a real environment in order to avoid being flagged. Additionally, some malware variants are built to take advantage of the blind spots in sandboxes. Sandboxing also creates some performance challenges; because it is time consuming, sandboxing every file is not possible.
Next-Gen Antivirus and Other Malware Detection Techniques and Blocking Methods
Heuristics and sandboxing are a good start to defeating malware, but considering their drawbacks, they are not enough. Rather, beating today’s modern malware requires newer, more dynamic tools. This need has led to the development of several NGAV (next-gen antivirus) methods. Let’s take a look at some of the most common ones.
AI/Machine-Learning-Based Static Analysis
Static analysis detection techniques that are based on machine learning train computers to recognize and differentiate between benign and malicious files. They teach the machine what’s good and what’s bad, so that eventually, the machine can sort the files on its own. These techniques take different behaviors (file behaviors, how long the file is open, traffic, everyday behavior, etc.), and calculate them into a conclusion on the nature of the file.
Although this is a very strong step in the right redirection, machine learning is not a perfect solution for detecting and defeating malware. In fact, machine learning techniques can be used in “adversarial attacks,” wherein attackers train machine learning systems to misclassify malware samples as non-malicious. Moreover, machine learning is still relatively new. As a malware solution, it’s still not particularly robust, so it needs to be used in conjunction with other tools and techniques.
While signatures are considered blacklisting, another approach to blocking malware is whitelisting. Whitelisting validates and controls all aspects of what a process is allowed to do, and blocks applications from doing everything—except what they’re supposed to do. This is beneficial for blocking threats such as zero-days, yet can be very frustrating for end users who want to run completely safe and legitimate applications, but are unable to. This is why application whitelisting should be used in high-risk environments and in kiosks like ATMs.
It should be noted that “standard” whitelisting relates to applications (allowed\not allowed), while “smart” whitelisting relates to processes within a legitimate app. For example, Microsoft Word is whitelisted, but the WINWORD.exe process is not allowed to request kernel handles to other processes.
Endpoint Detection and Response
Endpoint Detection and Response (EDR) monitors and records data and events from endpoint logs and packets. The collected data is analyzed to see what happens after infection, to look for IOCs to known malware campaigns, and to help organizations identify and respond to attacks.
Effective Malware Protection
Cynet 360 delivers full protection against the ever-changing malware landscape by continuously monitoring file execution and process behavior. Cynet’s malware protection comprises multiple complementing layers: signatures, ML-based static analysis, sandbox, analysis of process behavior in run time, and monitoring of critical memory locations. Cynet integrates the best of NGAV and EDR capabilities to disarm malware before it can fully execute and cause harm.
Attack methods are always changing, and there’s really nothing anyone can do to prevent attackers from using new tricks and methodologies. The key to beating these complex and evolving attacks is using a solution with many interlocking protection layers that detect and deflect them. Cynet’s breach protection platform is all the protection you need to defeat the nuanced attacks we’re up against.