Many OT engineers still believe that active scanning is not safe in OT environments. However, their assumptions don’t have a legitimate basis. 

Yes, regular network and vulnerability scanners can cause devices to act erratically. Printers start spewing out pages. Embedded systems freeze up or reboot. But it doesn’t have to be this way. If you observe a few key aspects and use a purpose-built scanner, actively detecting ICS and IoT equipment is entirely safe. runZero has proven that active scanning is safe, and it’s evident across numerous industries.

Digging into issues with legacy scanners

To better understand the challenges of active scanning, we analyzed why legacy vulnerability and network scanners destabilize systems. We found four different root causes:

• Malformed IP traffic
• Security probes
• Heavy scan traffic per device
• Snowflake devices

Let’s dig into each issue.

Malformed IP traffic

Legacy scanners often send intentionally malformed IP traffic to identify different flavors of operating systems. A robust TCP/IP stack on a Windows or Linux system will process the malformed traffic and respond in a specific manner that helps the scanner identify the flavor of the operating system.

Embedded systems often use legacy or custom TCP/IP stacks. When scanned with malformed IP traffic, these devices can freeze up or reboot because the unexpected traffic causes errors that are handled incorrectly by the stack.

Security probes

Vulnerability scanners send security probes, such as SQL injection exploits, to detect vulnerabilities in target systems. Embedded systems are often written without enough error handling built in, so the problem is similar as with malformed IP traffic: receiving unexpected network traffic can cause the devices to react erratically.

Heavy scan traffic per device

Legacy vulnerability and network scanners scan a large number of ports and can send several probes per port. This traffic is all sent to the end node in rapid succession. When all ports and probes are completed, the scanner moves on to the next host.

Enterprise IT hardware and mainstream operating systems can handle a lot of network traffic at once. OT equipment often doesn’t have a lot of processing power. Heavy scan traffic can overload the device, causing it to slow down or freeze up. In many industrial control applications, response times are critical. Even a slow down can have adverse effects on the overall environment.

Snowflake devices

When scanners avoid malformed IP traffic, security probes, and heavy scan traffic, most of the issues on OT networks can be resolved. However, there are a handful of particularly flakey devices that become unstable with even the most regular scan traffic. Serial-ethernet connectors, also known as print servers, tend to be among the worst “snowflake” devices.

Passive monitoring is expensive and lacks accuracy

That’s why by sticking with passive monitoring solutions instead of active scanning, OT engineers are inviting these issues into their projects:

• Longer deployment cycles – Connecting to SPAN ports or TAP appliances is more complex than deploying a software scanner in the environment.
• Higher cost – Requires lots of disk space and processing power, usually in the form of costly hardware appliances.
• Missing assets – You can’t inventory assets that are not communicating.
• Missing detail – Missing ports that are not communicating.
• Low accuracy – Spotty accuracy because passive monitoring is limited to analyzing existing traffic.
• Not future proof – The increasing amount of encrypted traffic makes passive monitoring solutions less viable over time.

Let’s take a look at the flip side and run through the key gains of leveraging an active scanning approach.

How to safely scan ICS environments

While legacy scanners cannot be used safely on OT assets, modern purpose-built scanners can safely scan ICS environments by following a few basic rules:

• Use only standard-conforming IP traffic – All traffic sent from the scanner must be completely RFC compliant.
• No security probes – Very easy. Just don’t use them.
• Throttle traffic per host – Limit the number of packets sent to each node. A good starting point is 40 packets per second. The best scanners keep overall scan times short by sending all traffic round-robin on the network when the threshold is reached.
• Probe for snowflakes – Detect snowflake devices before running a full port scan and adapt the scan for the particular model.

Now, let’s take a look at how these rules have been applied across different industries and what organizations have been able to uncover as a result.

Active scanning is a proven methodology across industries

Doing research in a lab is one thing, but proving a methodology in the field is another. This approach has been tested and deployed in production environments across many industries, including:

• Building automation
• Consumer and B2B electronics manufacturing
• Biomedical device manufacturing
• Telecommunications
• Broadcasting
• Universities (e.g., research instrumentation)
• Data center technology
• Transportation (e.g., train signals)
• City and state infrastructure (e.g., street signs, surveillance cameras)
• National labs
• Apparel manufacturing
• Car manufacturing
• Aerospace manufacturing
• Building material manufacturing
• Retail stores (e.g., POS systems, HVAC)
• Cattle and fish farms
• Utilities
• Saw mills
• Hospitals
• ICS equipment manufacturers

Some examples of equipment found in these environments include the following device types:

• PLCs
• Industrial control systems
• Serial-Ethernet converters
• HMI/HMI controllers/HDI
• BACNET devices
• Device servers
• Surveillance cameras
• Terminal servers
• Access control systems
• Intercoms
• KVMs
• Rugged WAP

Get started with active scanning of industrial control systems

You wouldn’t deploy a new piece of software across all of your devices without testing it first. The same is true for active scanning in ICS environments. As you’re considering rolling out active scanning technology, here are some tips to get you started:

1. Pick a purpose-built modern scanner – It’s unlikely that you will be successful with legacy network or vulnerability scanners as they send unsafe traffic. Pick a modern, purpose-built solution, such as runZero.
2. Start small and slow – If you have a small handful of devices in a lab, start there. Otherwise, pick a handful of devices to scan during a maintenance window and check their operational status afterwards. If you know you have snowflake devices, include them in your first scan. If it doesn’t work for them, it won’t work for the full network. Start with a very low network scan frequency, such as 1,000 packets per second from the scanner and 20 packets per second per host.
3. Try a bigger segment – Once you are comfortable with a handful of devices, scan a larger network segment during a maintenance window.
4. Plan your deployment – Deploy one scanner per network segment. Don’t scan through any network devices that filter traffic, otherwise the accuracy of your results will be impacted. Don’t scan through stateful devices because each IP/port connection will create another session and you may overload the device. Deploy the scanners on appropriate hardware or virtual machines. For a large network segment, you may want a dedicated host. For a medium-sized network, you can use an existing host. For small environments, you can even use a Raspberry Pi.

Hopefully, these tips will help you eradicate outdated and inaccurate perceptions against active scanning. Utilize these recommended best practices and you’ll be able to safely detect ICS and IoT devices via active scanning. runZero continues to prove this over and over again across multiple industries.

Printer manufacturer Lexmark recently published details on a vulnerability that affects over 100 of their printer models. Discovered by researcher Peter Geissler, this vulnerability can be leveraged to achieve unauthenticated remote code execution for an attacker. Firmware across devices in Lexmark’s small/medium business product line and also their enterprise product line have been found to contain this vulnerability.

What is the impact?

Lexmark assigned a CVSS score of 9.0 (“critical” severity rating) to this vulnerability (tracked as CVE-2023-23560), which allows server-side request forgery (SSRF) via the Web Services feature listening on port 65002 of affected printers. A successful attacker can exploit this vuln in a chain to gain code execution as root on vulnerable devices. Lexmark’s advisory states that, as of last week, they are not aware of anyone currently exploiting this vulnerability, but proof-of-concept exploit code is publicly available.

Are updates available?

All firmware versions (release numbers 081.233 and prior) for affected printer models contain this vulnerability (CVE-2023-23560). Lexmark has made firmware updates available for each affected device, via release numbers 081.234 and later (see Lexmark’s advisory for specific release version details per affected printer).

If updating firmware isn’t a near-term option for admins/owners of affected printers, Lexmark does offer a straightforward mitigation:

Disabling the Web-Services service on the printer (TCP port 65002) blocks the ability to exploit this
vulnerability. The port can be blocked by following process: “Settings”->“Network/Ports”- > “TCP/IP”- > “TCP/IP Port Access” then uncheck “TCP 65002 (WSD Print Service )” and save.

How do I find potentially vulnerable Lexmark printer assets with runZero?

Please note that the following query relies on you having already performed a scan with our latest Explorer/scanner release (v3.4.22), which now includes the scanning of port 65002. Alternatively, you can perform a new scan using an older Explorer/scanner, just add port 65002 to the Included TCP ports list under the Advanced tab of your task settings prior to running the scan.

From the Asset Inventory, use the following pre-built query to locate Lexmark printer assets which may need remediation:

type:printer AND vendor:Lexmark AND tcp_port:65002

Query results can then be checked against Lexmark’s list of vulnerable models and firmware versions.

As always, any prebuilt queries are available from our Queries Library. Check out the library for other useful inventory queries.