Introduction

Do an exercise, ask five IT technicians -of any profile- what SNMP means.. If you’re close with them the better, so that the first thing they do is not go to Wikipedia to boast. Hopefully, they might tell you what they said to me when I was working in networks.

“Security is Not My Problem”

Taking into account that the SNMP protocol is one of the monitoring bases, and a system that has been in use for more than thirty years, this answer, “Security is Not My Problem”, sums up the current monitoring situation quite well: ignorance, laziness and lack of interest in monitoring security.

By the way, we talked about SNMP in another article on our blog and I will give you a teaser in advance, it means Simple Network Management Protocol and it comes from 1987.

Considering that monitoring is “key to the kingdom”, since it allows access to all systems and even access many times with administration credentials, shouldn’t we take security a little more seriously when we talk about it?

Recent vulnerabilities in well-known monitoring systems such as Solarwinds or Centreon make the need to take security seriously in the implementation of monitoring systems increasingly urgent, since these have a very strong integration with systems.

In many cases, security problems are not so much about one piece of software being much safer than another, but about poor configuration and/or architecture. It must be taken into account that a monitoring system is complex, extensive, and in general, it is highly adapted to each organization. Today it was Solarwinds, tomorrow it could be Pandora FMS or Nagios.

No application is 100% secure, nor is any corporate network secured against intrusion, whatever the type. This is an increasingly evident fact and the only thing that can be done about it is to know the risks and assume which ones you can take, which ones absolutely not, and work on the latter.

Safe monitoring architecture

It is essential to keep in mind at all times that a monitoring system contains key information for a possible intruder. If monitoring falls into the wrong hands, your system will be compromised. That is why it is so important to devote time to the architecture of your monitoring system, whatever it may be.

Carry out a first analysis, collecting the requirements and scope of your monitoring strategy:

• Identify what systems you are going to monitor and catalogue their security levels.
• Identify which profiles will have access to the monitoring system.
• Identify how you will obtain information from those systems, whether through probes/agents or remote data.
• Identify who is responsible for the systems you are going to monitor.

The architecture of a system will have, whatever the chosen software, the following elements and will have to take into account its network topology, its resources and the way to protect them properly:

1. Information display interface (web console, heavy application).
2. Data storage (usually a relational database).
3. Information collectors (intermediate servers, pollers, collectors, etc.).
4. Agents (optional).
5. Notification system (alerts, notices, etc.).

Monitoring system securing

No matter how correct the implementation of a system, its architecture and its design as a whole is, if one of the elements that make it up is violated, the damage it may suffer by a malicious attack compromises the entire structure. For this reason, in security there is a saying, “Security is a chain and your real security always depends on its weakest point.”

This list of security concepts applied to the architecture of a monitoring system can be summarized as the features that a monitoring product must have to ensure maximum security in an implementation:

• Encrypted traffic between all its components.
• High availability of all its components.
• Integrated backup.
• Double access authentication.
• Delegated authentication system (LDAP, AD, SAML, Kerberos, etc.).
• ACL and user profiling.
• Internal audit.
• Password policy.
• Sensitive data encryption.
• Credential containers.
• Monitoring of restricted areas/indirect access.
• Installation without superuser.
• Safe agent/server architecture (passive).
• Centralized and distributed update system.
• 24/7 support.
• Clear vulnerability management policy by the manufacturer.

Monitoring infrastructure basic securing

The management console, monitoring servers and other elements should never be on an accessible public network. The console should always be protected on an internal network, protected by firewalls and, if possible, on a network independent from other management systems.

The operating systems that host the monitoring infrastructure should not be used for other purposes: for example, to reuse the database for other applications, nor the base operating systems to run other applications.

Safe and encrypted traffic

You should make sure that your system supports SSL/TLS encryption and certificates at both ends at all levels: user operation, communication between components or sending data from the agent to the servers.

If you are going to use agents in unsafe locations, it is highly recommended that you force all external agents to use certificate-based authentication at both ends, to avoid receiving information from unauthorized sources and to prevent information collected by agents to not travel transparently.

On the other hand, it is very important for you to activate encryption on your web server to provide an encrypted administration console and prevent any attacker from seeing access credentials, remote system passwords or confidential information.

Full High Availability

For all elements: database, servers, agents and console.

Integrated backup

The tool itself should make this as easy as possible, as settings and data are often highly distributed and consistent backup is complex.

Clear vulnerability management policy by the manufacturer

Every day, dozens of independent auditors test the strengths and weaknesses of all kinds of business applications. They seek to gain a foothold in the sector by publishing an unknown ruling to increase their reputation. Many clients, as part of their internal security management processes, execute external and internal security audits that target their IT infrastructure.

Be that as it may, all products have security flaws, the question is: how are those flaws handled? Transparency, diligence and communication are essential to prevent customers from having problems derived from vulnerabilities in the software they use. It is essential that there is a clear policy in this regard, so that it is known which public vulnerabilities have been reported, when they have been corrected and if a new one is detected, the steps to follow for notification, mitigation and distribution to the end customer.

Dual authentication system

Pandora FMS has an -optional- system based on google authenticator that allows forcing its use for all users for security policies. This will make user access to the administration console much safer, preventing that due to privilege escalation the system can be accessed as administrator, which is, at best, the highest risk that can be run.

Delegated authentication system

Complementary to the previous one, you can delegate management console authentication to authenticate against LDAP, Active Directory, or SAML. It will enable a centralized access management, and combined with the double authentication system your access will become much safer.

ACL and user profiling

Identify and assign different users to specific people. Do not use generic users, assign only the necessary permissions and do not use “super administrators”. They are good practices not only for monitoring tools but for any business software implementation with access to sensitive information.

Nowadays, any professional tool to define an access profile for each user will do so in such a way that no user has “absolute control”, but only has the minimum required access to their functions.

Internal audit system

You must have a system in place to record all user actions, including information on altered or deleted fields. Said system must be able to be exported abroad so that not even the administrator user can alter said records.

Password policy

A basic element that allows you to enforce a strict password management policy for access to application users: minimum password size, password type, their reuse, forced change once in a while, etc.

Sensitive data encryption

The system must allow the most sensitive data to be stored encrypted and safely, such as access credentials, monitoring element custom fields, etc. Even if the system itself contains the encryption “seed”, it will always be much more difficult for a potential attacker to access this information.

Credential containers

Or an equivalent system for the administrator to delegate credential use to other users who use said credentials to monitor elements without seeing the passwords contained in the container.

Restricted area monitoring

In these systems, information will be collected remotely by a satellite server and will be available to be collected from the central system (in Pandora FMS through a specific component called Sync server). That way, data can be collected from a network without access to the outside, ideal for very restrictive environments where the impact is drastically reduced if an attacker takes over the system.

Agent remote management locking system

For critical security environments, where the agent cannot be remotely managed once it is configured. This is especially critical in monitoring, since if a system is compromised and its administration is accessed, by the way the system is configured itself, it will have access to all systems from where it receives information. In critical systems, the remote management capacity must be deactivated, even if that makes administration more tricky. The same applies to automatic updates on the agent.

Design of safe architecture for communication with agents

Sometimes known as passive communication. That way, agents will not listen to a port nor have remote access from the console. They are the ones who will connect to the central system to ask for instructions.

Installation without root

Pandora FMS can be installed in environments with custom paths without running with root. In some banking environments, it is a requirement that we meet.

Notification and reporting system (alerts, notices, etc.)

A monitoring system is only useful if it shows accurate information when it is needed. Alert or weekly report reception is the culmination of all the previous work and for that you will have to take into account some “obvious” points that are often overlooked. Protect those systems, wherever they may be.

Periodic updates

All manufacturers now distribute regular updates, which include both bug fixes and security problems. In our case, we publish updates approximately every five weeks. It is essential to update systems as soon as possible, because when a vulnerability is reported, product managers ask external security researchers who have reported the bug, not to publish anything about the vulnerability until a patch is published. Once the patch is published, the researcher will publish the information in more detail as wished, a fact that can be used to exploit and attack non-updated software versions.

Pandora FMS has a vulnerability disclosure public policy as well as a public catalog of known and reported vulnerabilities. Our policy has maximum transparency and full communication with security researchers, always to mitigate the impact of any security problem and to be able to protect our clients as a top priority.

24/7 support

In our support, the technician who answers the phone has the whole team backing him up. If there is a security issue and a security patch has to be published within hours. We not only have the technology to spread the patch to all our customers, but also the team to develop it in record time.

Base system securing

Hardening or system securing is a key point in the global security strategy of a company. As manufacturers, we issue a series of recommendations to carry out a safe installation of all Pandora FMS components, based on a standard RHEL7 platform or its equivalent Centos7. These same recommendations are valid for any other monitoring system:

Hardening checklist for monitoring base system:

• System access credentials.
• Superuser access management.
• System access audit.
• SSH securing.
• Web server securing.
• DB server securing.
• Server minimization.
• Local monitoring.

Access credentials

To access the system, nominative access users will be created, without privileges and with access restricted to their needs. Ideally, the authentication of each user should be integrated with a double authentication system, based on token. There are free and safe alternatives such as Google Authenticator that can be easily integrated into Linux, although outside the scope of this guide. Seriously consider its use.

If it is necessary to create other users for applications, they must be users without remote access (for this, it is necessary to deactivate their Shell or some equivalent method).

Superuser access through sudo

In the event that certain users must have administrator permissions, SUDO will be used.

Base system access audit

It is necessary to have the security log /var/log/secure active and monitor those logs with monitoring (which we will see later).

By default CentOS has this enabled. If not, just check the /etc/rsyslog.conf or /etc/syslog.conf file.

We recommend you to take the logs from the audit system and collect them with an external log management system. Pandora FMS can do it easily and it will be useful to set alerts or review them centrally in case of need.

SSH server securing

The SSH server allows you to remotely connect to your Linux systems to execute commands, so it is a critical point and must be secured by paying attention to the following points:

• Modify default port.
• Disable root login.
• Disable port forwarding.
• Disable tunneling.
• Remove SSH keys for remote root access.
• Investigate the source of keys for remote access. To do this, look at the content of the file /home/xxxx/.ssh/authorized_keys and see which machines they are from. Delete them if you think there shouldn’t be any.
• Establish a standard remote access banner that clearly explains that the server is a private access server and that anyone without credentials should log out.

MySQL server securing

Listening port. If MySQL server has to provide service to the outside, just check that the root credentials are safe. If MySQL only gives service to an internal element, make sure that it only listens on localhost.

Web server securing

We will modify the configuration to hide the Apache and OS version in the server information headers.

If you use SSL, disable unsafe methods. We recommend the use of TLS 1.3 only.

System service minimizing

This technique can be very exhaustive. It consists simply of eliminating everything that is not necessary in the system. Thus we avoid possible problems in the future with poorly configured applications that we really did not need and that can be vulnerable in the future.

Local monitoring

All the internal monitoring systems would have to be monitored to the highest level, specially information registries. In our case the following active controls in addition to the standard controls are always recommended:

• Active security Plugin.
• Complete system inventory (specially users and installed packages).
• System logs and server security.