Incident Overview
On October 7, the email server of a big commercial pharma organization was attacked. It was running Zimbra 8.x version on CentOS and got quickly compromised. Malicious actor exploited Internet-facing Zimbra Collaboration Suite using CVE-2022-41352 “cpio” zero-day vulnerability.
Our investigation revealed and was able to confirm that unknown APT groups are massively exploiting an unpatched vulnerability (CVE-2022-41352) in Zimbra Collaboration Suite to infect vulnerable servers.
Initial foothold was discovered through CrowdStrike EDR on that Linux mail server unfortunately for the sutomer it just detected but did not prevent the exploitation because of the insufficient Prevention Policy aggressiveness as customer had just started dealing with Crowdstrike software and it was in fine-tuning mode. Soon after the detection, MDR/SOC team initiated Incident Response, gathered information and contacted the client’s representatives via Google Meet.
After the approval, the host was network-isolated, all client’s endpoints were moved to the highest Prevention Policy.
Recommendations & Remediation
Since Zimbra released a patch for this vulnerability, the best course of action is to update your devices immediately. If this, for some reason, is not possible, installing pax on the machine hosting the Zimbra installation will prevent the vulnerability from being exploitable. pax is available from package managers (such as apt and yum) of all major Linux distributions. Among all Linux variants officially supported by Zimbra, only Ubuntu installs pax by default and is therefore not affected by CVE-2022-41352:
| Distribution | Vulnerable to CVE-2022-41352 |
| Red Hat Enterprise Linux 7 | Yes |
| Red Hat Enterprise Linux 8 | Yes |
| CentOS 7 | Yes |
| CentOS 8 | Yes |
| Oracle Linux 7 | Yes |
| Oracle Linux 8 | Yes |
| Rocky Linux 8 | Yes |
| Ubuntu 16.04 LTS | No |
| Ubuntu 18.04 LTS | No |
| Ubuntu 20.04 LTS | No |
Please note that installing pax doesn’t address the root issue with any distribution, where other program paths, both within and outside of Zimbra could still cause cpio to process untrusted data.
After taking the aforementioned mitigation steps, owners of Zimbra servers are encouraged to check for traces of compromise. The following paths are known locations for webshells deployed by malicious actors currently leveraging CVE-2022-41352:
| 123456 | /opt/zimbra/jetty/webapps/zimbra/public/.error.jsp/opt/zimbra/jetty/webapps/zimbra/public/ResourcesVerificaton.jsp/opt/zimbra/jetty/webapps/zimbra/public/ResourceVerificaton.jsp/opt/zimbra/jetty/webapps/zimbra/public/ZimletCore.jsp/opt/zimbra/jetty/webapps/zimbra/public/searchx.jsp/opt/zimbra/jetty/webapps/zimbra/public/seachx.jsp |
In addition, it is worth noting that the Metasploit exploit drops its webshell in the following location:
| 1 | /opt/zimbra/jetty_base/webapps/zimbra/[4-10 random characters].jsp |
If you discover one of these files on your Zimbra installation, please contact an incident response specialist as soon as possible. Removing the file is not enough. Performing disinfection on Zimbra is extremely difficult, as the attacker will have had access to configuration files containing passwords used by various service accounts. These credentials can be used to regain access to the server if the administrative panel is accessible from the internet. In addition, considering the rudimentary nature of all webshells we have discovered so far, it is almost certain that attackers will deploy more robust and sophisticated backdoors as soon as they get the chance.
Case Details
October 7, mail server running Zimbra 8.x version on CentOS was compromised. An unidentified malicious actor exploited Internet-facing Zimbra Collaboration Suite using CVE-2022-41352 “cpio” zero-day vulnerability.
After the approval, the host was network-isolated, all client’s endpoints were moved to the highest Prevention Policy.
Based on the CrowdStrike data, IR team stopped the threat quickly enough and did not find any traces of data exfiltration, confidential data access or any activity that may severely harm business continuity.
Initial Access
On October 7, at 15:55, malicious actor achieved RCE with root permissions via Zimbra vulnerability.The server infection began from downloading a few unknown files, most notably /tmp/.opt/sh. Content and purpose of other files are unknown, as well as the reason for the new Nginx server execution.
Execution & Reconnaissance
A few seconds later after the download, malicious actor executed an unknown file: /opt/zimbra/common/libexec/slapd -u root -g root -f /tmp/.opt/cfg2
Immediately after, the main infection script was started. Its purpose was to deeply persist on the system.
Persistence & Defense Evasion
As seen from the CrowdStrike process timeline, the malicious script was executed successfully, given the wget command to the attackers host with status=0 URL parameter.
“Zero” status indicates successful persistence via SSH key, which will be proven a few slides later.
Main Script: Bash copy with SUID
The hacker copied the Bash binary to /usr/lib/sftp in order to avoid detection via logs analysis.
To make it work, new Bash copy was made executable and was given SUID flag to run as root.
The last trick to avoid detection was to use touch -r command and copy modification time from passwd to the newly created backdoor.
Main Script: Web shell & SSH key
The web shell part seems to be incorrect. It does not create a new one, but only renames the old web shell, if exists. According to our analysis, old web shell did not exist during the script execution.
Then, the hacker prepares a few reusable functions and creates new SSH key pair. It was meant to be used for SSH backdoor on root user.
Main Script: SSH persistence details
The adversary verified many sshd file configurations to ensure that backdoored key will work as expected.
Furthermore, the hacker tuned firewall rules and changed default root shell to guarantee correct exploitation.
As a part of defense evasion, touch -d command was used to revert original modification time of the affected sshd config.
Main Script: Cleanup and callback
The last action the script performed was to notify the attacker-controlled server about the end of exploitation. As was shown in the logs before, the callback status is zero, which means success.
Finally, the hacker patched the exploited CVE simply by removing the vulnerable cpio package and restarting the Zimbra service. Since the CVE is actively exploited, this was the logical step to avoid conflicts with other hackers’ activities while retaining a solid persistence on the host.
Main Script: Attachments
The SOC team has not detected any further suspicious SSH logins, exfiltration attempts or other suspicious actions on all of the monitored client’s devices.
Still, the client has provided multiple proofs of successful exploitation. Clockwise, the screenshots are:
- New sftp binary with the same size as bash, having SUID flag and modified timestamp
- Infected .ssh folder containing hacker’s keys and modified authorized_keys file
- SSH backdoor in authorized_keys file, the same as in the presented script
Customer reaction:
Based on the CrowdStrike data, UnderDefense Incident Response team stopped the threat quickly enough before any negative consequences. Incident Response did not find any traces of data exfiltration, confidential data access or any activity that might have severely harmed business continuity.
The client chose to go with a complete migration to a clean server and copy the Zimbra database and its configs to the new server. Given that fact, it is highly recommended to:
- Install the newest Zimbra version, and mitigate the vulnerability as described here
- Ensure Zimbra is installed from a service, low-privileged user, not root account
- Provide file integrity check on all Zimbra files to eliminate .jsp backdoors
- Reset passwords for the infected server users, enable MFA for Zimbra admins
About Version 2
Version 2 is one of the most dynamic IT companies in Asia. The company develops and distributes IT products for Internet and IP-based networks, including communication systems, Internet software, security, network, and media products. Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.
About UnderDefense
UnderDefense, a globally top-ranked firm by Gartner and Clutch, provides cyber resiliency consulting and technology-enabled services to anticipate, manage and defend against cyber threats. We empower clients to predict, prevent, detect, and respond to threats.
