ESET’s journey through 2024’s TTP-heavy Enterprise ATT &CK evaluation demonstrates both progress and scope for continued enhancement of our detection-and-response solution, as well as an interesting contrast with commercial tests.

Six years into MITRE Enterprise evaluations, with their consistent focus on attack emulations leveraging a vast collection of tactics, techniques and procedures (TTPs), vendors are now finding evolved detection and protection scenarios added to the mix. MITRE’s team of experts has clearly been working agilely, prompting vendors to adapt to the new test environment. Simultaneously, vendors looking to position themselves for the “win” and improve sales prospects increasingly face a paradox with MITRE: It’s still not a competitive test. Rather, the takeaways provide most value to security analysts involved in the day-to-day work of operating endpoint detection and response platforms.

This year’s evaluation has introduced some substantial changes. Notably, MITRE dropped telemetry from the detection categories, raising the threshold for a recorded event to be considered a detection. To counteract vendors’ gaming products to “win” or “detect everything,” a proportion of substeps now test for false positives rather than detections, while other substeps are not evaluated at all. In a final nod to address “points” accumulated via “detecting everything,” MITRE measures the number of detections in the dashboard, again cutting into “point gain” via bloated numbers of detections.

ESET also brought changes to the table, most significantly by fielding ESET Incident Creator, a module in ESET Inspect for incident-focused handling of threats. Incident Creator proved its use case in the detection portion of the evaluation, transforming how analysts are presented with detections generated from a cyberattack; detections are rolled up into incidents, giving analysts a focused view of how each attack progressed. With this tool in hand, we demonstrated good visibility into each attack scenario, detecting all steps and most meaningful substeps while maintaining a low volume of detections. Thus, for ESET, most missed substeps, e.g., API calls, provide the analyst with little to no value, could be discovered via other means, and present almost no loss of information for the reconstruction of the attack.

The most recent round of the ATT&CK Evaluations Enterprise pitted multiple vendors’ detection and response platforms against three attack scenarios named DPRK, CI0p, and LockBit, as well as a battery of 10 protection tests. With well-recognized research on all of the threat groups chosen for the scenarios, ESET engineers were interested to observe how ESET Inspect’s behavior could translate to supporting SOC operators as well as organizations leveraging security service offers such as ESET MDR.

With a nod to ESET’s preoccupation with efficiency, and MITRE’s new rules seeking to limit a detect-everything approach, the three attacks generated plenty of detections – correlated into incidents – allowing incident responders to easily understand how the attacks happened and thus take quick remediation. Additionally, the Incident Creator module eliminated considerable noise for the benefit of threat analysts – another priority for ESET, both inside and outside this test environment.

With one of MITRE’s goals being to examine coverage of the TTPs in the ATT&CK knowledge base, vendors’ products in the detection scenarios are configured to not block or kill any of the threats they encounter. This is an important point because, in the real world, many of the threats employed in these scenarios would have been blocked either on first encounter or very soon thereafter.

For ESET, there are several fitting cases to highlight: The CI0p and LockBit ransomware samples, along with a couple other samples, would have been blocked in the first evaluation run. Although masked by the evaluation’s focus on ESET Inspect, it was actually ESET Live Guard Advanced, a key technology layer in the ESET PROTECT platform, that was behind the initial detection and blocking of these ransomware samples. Any remaining malware would have been blocked on the second run. This is good news in the real world because it means that customers benefit from efficient protection against threats.

In regard to the protection tests, while there are a few areas to improve the coverage provided by our products, 2024’s tests mainly focused on isolated and sometimes not clearly malicious substeps that, in the real world, could actually be benign and thus cause harm to customers if blocked. On real-world systems, cyberattackers aren’t running techniques and procedures “atomically,” i.e., disconnected from a full attack chain. Instead, their operations are a buildup, and the task of detection and response solutions is to correlate the full context to identify correctly that something malicious is afoot.

Figure 2. Detection of the SDBbot installer in the Cl0p config change run

Figure 3. Detection of the Firefox password stealer in the LockBit config change run

We are left feeling that the pursuit of total coverage of the ATT&CK knowledge base does not necessarily improve defenses or automatically assist security analysts in their daily work. In contrast to forensics or malware analysis, detection and response require only enough coverage of highly prevalent or severe techniques (or substeps) for the analyst to trigger a response. Missing detections for low-prevalence or low-severity techniques does not necessarily translate to lower protection. Quite the opposite, it could mean that work is streamlined, and remediation is faster because the major steps needed to identify the attack are immediately highlighted, thereby allowing a timely and adequate response to be triggered – in some cases even with automatic blocking of the detected threat.

Again, outside the ATT&CK Evaluations and atomic testing scenarios, third-party tests usually design scenarios that don’t prioritize coverage of TTPs. Instead, they measure blocking, false positives, performance, cost, and the like by a fully functioning product suite. They also focus more on critical steps of an attack, such as endpoint compromise and foothold, internal propagation, and asset breach.  The ultimate arbiter is stopping an attack, whether by blocking malicious files or killing malicious processes, or at least providing the security analyst with the means to do so.

A number of strong results demonstrate the efficacy of real-world performance. This includes real-world tests, such as AV-Comparatives’ Endpoint Prevention & Response (EPR) Test 2024 and 2023, and analyst perspectives, including selection of ESET as a leader in both KuppingerCole’s Leadership Compass for MDR and Leadership Compass for Endpoint Protection Detection & Response.

In KuppingerCole’s executive summary of the EPDR market, they discuss the wider industry’s use of the MITRE ATT&CK framework, as well as the fact that endpoint protection platforms and EDR tool sets have converged into the logical integrated pairing of Endpoint Protection Detection & Response (EPDR). This means the majority of vendors aim to improve based on MITRE’s micro-emulation of techniques and procedures, while developing successful, integrated EPDR proven by industry-standard, real-world testing. Ideally, covering these two scopes makes for battle-ready detection and response.

In fact, EPDR solutions are the battle-ready systems that institutions and businesses invest in, deploy, and operate today. Above and beyond this, ESET continues to seek out other avenues to challenge its products and its engineers. Perhaps the zenith of this search has been reached with our regular participation at Locked Shields, a military simulation of today’s digital battlespace organized by the NATO Cooperative Cyber Defense Centre of Excellence.

In 2024, more than 60 ESET system engineers, security monitoring analysts, malware researchers and analysts, and comms specialists formed ranks with defenders from the Slovak and Hungarian militaries and the private and academic sectors to defend our assigned battlespace.

Our tool, ESET Inspect, backed by our multilayered security platform ESET PROTECT, was central to Slovakia-Hungary’s fourth place out of 18 participating teams and helped the team to top three placings in cyber threat intelligence, client-side protection and forensics. This cyber-battle simulation was an intensively immersive experience for all the technologists involved, whether as threat analysts working to understand the TTPs to anticipate the next stages of an attack or engineers configuring cyber defenses.

ESET remains as motivated to leverage MITRE’s Enterprise Evaluations to feed technical audiences with critical food for thought as we are to invest our institutional knowledge in Locked Shields to advance both our security practice and that of our NATO partners. The output of our engagement with MITRE, whether via the +350 contributions to the ATT&CK knowledge base or participation in the ATT&CK Evaluations, continues to be sharpened detection and response capabilities tuned to the real world and protecting our shared progress.