Introduction

Do an exercise, ask five IT technicians -of any profile- what SNMP means.. If you’re close with them the better, so that the first thing they do is not go to Wikipedia to boast. Hopefully, they might tell you what they said to me when I was working in networks.

“Security is Not My Problem”

Taking into account that the SNMP protocol is one of the monitoring bases, and a system that has been in use for more than thirty years, this answer, “Security is Not My Problem”, sums up the current monitoring situation quite well: ignorance, laziness and lack of interest in monitoring security.

By the way, we talked about SNMP in another article on our blog and I will give you a teaser in advance, it means Simple Network Management Protocol and it comes from 1987.

Considering that monitoring is “key to the kingdom”, since it allows access to all systems and even access many times with administration credentials, shouldn’t we take security a little more seriously when we talk about it?

Recent vulnerabilities in well-known monitoring systems such as Solarwinds or Centreon make the need to take security seriously in the implementation of monitoring systems increasingly urgent, since these have a very strong integration with systems.

In many cases, security problems are not so much about one piece of software being much safer than another, but about poor configuration and/or architecture. It must be taken into account that a monitoring system is complex, extensive, and in general, it is highly adapted to each organization. Today it was Solarwinds, tomorrow it could be Pandora FMS or Nagios.

No application is 100% secure, nor is any corporate network secured against intrusion, whatever the type. This is an increasingly evident fact and the only thing that can be done about it is to know the risks and assume which ones you can take, which ones absolutely not, and work on the latter.

Safe monitoring architecture

It is essential to keep in mind at all times that a monitoring system contains key information for a possible intruder. If monitoring falls into the wrong hands, your system will be compromised. That is why it is so important to devote time to the architecture of your monitoring system, whatever it may be.

Carry out a first analysis, collecting the requirements and scope of your monitoring strategy:

• Identify what systems you are going to monitor and catalogue their security levels.
• Identify which profiles will have access to the monitoring system.
• Identify how you will obtain information from those systems, whether through probes/agents or remote data.
• Identify who is responsible for the systems you are going to monitor.

The architecture of a system will have, whatever the chosen software, the following elements and will have to take into account its network topology, its resources and the way to protect them properly:

1. Information display interface (web console, heavy application).
2. Data storage (usually a relational database).
3. Information collectors (intermediate servers, pollers, collectors, etc.).
4. Agents (optional).
5. Notification system (alerts, notices, etc.).

Monitoring system securing

No matter how correct the implementation of a system, its architecture and its design as a whole is, if one of the elements that make it up is violated, the damage it may suffer by a malicious attack compromises the entire structure. For this reason, in security there is a saying, “Security is a chain and your real security always depends on its weakest point.”

This list of security concepts applied to the architecture of a monitoring system can be summarized as the features that a monitoring product must have to ensure maximum security in an implementation:

• Encrypted traffic between all its components.
• High availability of all its components.
• Integrated backup.
• Double access authentication.
• Delegated authentication system (LDAP, AD, SAML, Kerberos, etc.).
• ACL and user profiling.
• Internal audit.
• Password policy.
• Sensitive data encryption.
• Credential containers.
• Monitoring of restricted areas/indirect access.
• Installation without superuser.
• Safe agent/server architecture (passive).
• Centralized and distributed update system.
• 24/7 support.
• Clear vulnerability management policy by the manufacturer.

Monitoring infrastructure basic securing

The management console, monitoring servers and other elements should never be on an accessible public network. The console should always be protected on an internal network, protected by firewalls and, if possible, on a network independent from other management systems.

The operating systems that host the monitoring infrastructure should not be used for other purposes: for example, to reuse the database for other applications, nor the base operating systems to run other applications.

Safe and encrypted traffic

You should make sure that your system supports SSL/TLS encryption and certificates at both ends at all levels: user operation, communication between components or sending data from the agent to the servers.

If you are going to use agents in unsafe locations, it is highly recommended that you force all external agents to use certificate-based authentication at both ends, to avoid receiving information from unauthorized sources and to prevent information collected by agents to not travel transparently.

On the other hand, it is very important for you to activate encryption on your web server to provide an encrypted administration console and prevent any attacker from seeing access credentials, remote system passwords or confidential information.

Full High Availability

For all elements: database, servers, agents and console.

Integrated backup

The tool itself should make this as easy as possible, as settings and data are often highly distributed and consistent backup is complex.

Clear vulnerability management policy by the manufacturer

Every day, dozens of independent auditors test the strengths and weaknesses of all kinds of business applications. They seek to gain a foothold in the sector by publishing an unknown ruling to increase their reputation. Many clients, as part of their internal security management processes, execute external and internal security audits that target their IT infrastructure.

Be that as it may, all products have security flaws, the question is: how are those flaws handled? Transparency, diligence and communication are essential to prevent customers from having problems derived from vulnerabilities in the software they use. It is essential that there is a clear policy in this regard, so that it is known which public vulnerabilities have been reported, when they have been corrected and if a new one is detected, the steps to follow for notification, mitigation and distribution to the end customer.

Dual authentication system

Pandora FMS has an -optional- system based on google authenticator that allows forcing its use for all users for security policies. This will make user access to the administration console much safer, preventing that due to privilege escalation the system can be accessed as administrator, which is, at best, the highest risk that can be run.

Delegated authentication system

Complementary to the previous one, you can delegate management console authentication to authenticate against LDAP, Active Directory, or SAML. It will enable a centralized access management, and combined with the double authentication system your access will become much safer.

ACL and user profiling

Identify and assign different users to specific people. Do not use generic users, assign only the necessary permissions and do not use “super administrators”. They are good practices not only for monitoring tools but for any business software implementation with access to sensitive information.

Nowadays, any professional tool to define an access profile for each user will do so in such a way that no user has “absolute control”, but only has the minimum required access to their functions.

Internal audit system

You must have a system in place to record all user actions, including information on altered or deleted fields. Said system must be able to be exported abroad so that not even the administrator user can alter said records.

Password policy

A basic element that allows you to enforce a strict password management policy for access to application users: minimum password size, password type, their reuse, forced change once in a while, etc.

Sensitive data encryption

The system must allow the most sensitive data to be stored encrypted and safely, such as access credentials, monitoring element custom fields, etc. Even if the system itself contains the encryption “seed”, it will always be much more difficult for a potential attacker to access this information.

Credential containers

Or an equivalent system for the administrator to delegate credential use to other users who use said credentials to monitor elements without seeing the passwords contained in the container.

Restricted area monitoring

In these systems, information will be collected remotely by a satellite server and will be available to be collected from the central system (in Pandora FMS through a specific component called Sync server). That way, data can be collected from a network without access to the outside, ideal for very restrictive environments where the impact is drastically reduced if an attacker takes over the system.

Agent remote management locking system

For critical security environments, where the agent cannot be remotely managed once it is configured. This is especially critical in monitoring, since if a system is compromised and its administration is accessed, by the way the system is configured itself, it will have access to all systems from where it receives information. In critical systems, the remote management capacity must be deactivated, even if that makes administration more tricky. The same applies to automatic updates on the agent.

Design of safe architecture for communication with agents

Sometimes known as passive communication. That way, agents will not listen to a port nor have remote access from the console. They are the ones who will connect to the central system to ask for instructions.

Installation without root

Pandora FMS can be installed in environments with custom paths without running with root. In some banking environments, it is a requirement that we meet.

Notification and reporting system (alerts, notices, etc.)

A monitoring system is only useful if it shows accurate information when it is needed. Alert or weekly report reception is the culmination of all the previous work and for that you will have to take into account some “obvious” points that are often overlooked. Protect those systems, wherever they may be.

Periodic updates

All manufacturers now distribute regular updates, which include both bug fixes and security problems. In our case, we publish updates approximately every five weeks. It is essential to update systems as soon as possible, because when a vulnerability is reported, product managers ask external security researchers who have reported the bug, not to publish anything about the vulnerability until a patch is published. Once the patch is published, the researcher will publish the information in more detail as wished, a fact that can be used to exploit and attack non-updated software versions.

Pandora FMS has a vulnerability disclosure public policy as well as a public catalog of known and reported vulnerabilities. Our policy has maximum transparency and full communication with security researchers, always to mitigate the impact of any security problem and to be able to protect our clients as a top priority.

24/7 support

In our support, the technician who answers the phone has the whole team backing him up. If there is a security issue and a security patch has to be published within hours. We not only have the technology to spread the patch to all our customers, but also the team to develop it in record time.

Base system securing

Hardening or system securing is a key point in the global security strategy of a company. As manufacturers, we issue a series of recommendations to carry out a safe installation of all Pandora FMS components, based on a standard RHEL7 platform or its equivalent Centos7. These same recommendations are valid for any other monitoring system:

Hardening checklist for monitoring base system:

• System access credentials.
• Superuser access management.
• System access audit.
• SSH securing.
• Web server securing.
• DB server securing.
• Server minimization.
• Local monitoring.

Access credentials

To access the system, nominative access users will be created, without privileges and with access restricted to their needs. Ideally, the authentication of each user should be integrated with a double authentication system, based on token. There are free and safe alternatives such as Google Authenticator that can be easily integrated into Linux, although outside the scope of this guide. Seriously consider its use.

If it is necessary to create other users for applications, they must be users without remote access (for this, it is necessary to deactivate their Shell or some equivalent method).

Superuser access through sudo

In the event that certain users must have administrator permissions, SUDO will be used.

Base system access audit

It is necessary to have the security log /var/log/secure active and monitor those logs with monitoring (which we will see later).

By default CentOS has this enabled. If not, just check the /etc/rsyslog.conf or /etc/syslog.conf file.

We recommend you to take the logs from the audit system and collect them with an external log management system. Pandora FMS can do it easily and it will be useful to set alerts or review them centrally in case of need.

SSH server securing

The SSH server allows you to remotely connect to your Linux systems to execute commands, so it is a critical point and must be secured by paying attention to the following points:

• Modify default port.
• Disable root login.
• Disable port forwarding.
• Disable tunneling.
• Remove SSH keys for remote root access.
• Investigate the source of keys for remote access. To do this, look at the content of the file /home/xxxx/.ssh/authorized_keys and see which machines they are from. Delete them if you think there shouldn’t be any.
• Establish a standard remote access banner that clearly explains that the server is a private access server and that anyone without credentials should log out.

MySQL server securing

Listening port. If MySQL server has to provide service to the outside, just check that the root credentials are safe. If MySQL only gives service to an internal element, make sure that it only listens on localhost.

Web server securing

We will modify the configuration to hide the Apache and OS version in the server information headers.

If you use SSL, disable unsafe methods. We recommend the use of TLS 1.3 only.

System service minimizing

This technique can be very exhaustive. It consists simply of eliminating everything that is not necessary in the system. Thus we avoid possible problems in the future with poorly configured applications that we really did not need and that can be vulnerable in the future.

Local monitoring

All the internal monitoring systems would have to be monitored to the highest level, specially information registries. In our case the following active controls in addition to the standard controls are always recommended:

• Active security Plugin.
• Complete system inventory (specially users and installed packages).
• System logs and server security.

Pandora FMS started as a totally personal open source project back in 2004. I wasn’t even a professional programmer, I was doing Unix security consulting. In fact, I chose PHP but Pandora FMS was my first application with PHP, I knew some things about ASP and my favorite programming language had been C.

A project with a single programmer and no professional users of his software yet is very different from a project with several dozen programmers and hundreds of clients using the software in critical environments. The evolution that Pandora FMS has undergone from 2004 to 2021 is a real case of steady improvement in software engineering.

Fortunately, I did not pay much attention to that subject of the degree, because most of the things that work and that I have learned with practice do not come in a book, nor are they explained at the university, because each software project and each team of people is very different. It may sound cliché, but it is the truth, and it is better to accept it and avoid formulas, because building a solid software product that can grow over time is not trivial at all.

In this article, I am going to talk about our experience, our evolution over time, but above all, about how our engineering processes work today. I have always believed that the most important part of open source is transparency, and that this should apply to everything, not only to software but also to processes and knowledge in general.

Version control system

It is an essential part of any software project. Today the ubiquitous GIT is everywhere (by the way, not everyone knows that Git is the work of Linus Torvalds, original author of the Linux kernel). A version control system helps, in short, a group of developers work without overlapping their jobs.

When the Pandora FMS project started, I was working without version control, because there were no other people. When some people began to collaborate on it, we realized that a simple shared directory was not worth it, because we were overlapping the code and, yes, making backups to save old versions was not a very efficient method.

The first version control system we used was CVS, which we have been using for eight years or more. Around 2008, we started using SVN (Subversion) another slightly more efficient system and it wasn’t until 2013 when we started using GIT and opened our official repository on Github.

Pandora FMS public repository on Github

Since Pandora FMS has an open source version and an Enterprise version -with proprietary code and commercial licenses- we have two GIT projects, one public on GitHub and the other private, which we manage with GitLab. The GitHub version is in sync with our private copy on GitLab at our offices. Some partners who collaborate with us in developing have access to this private repository, and through an extension of our support application (Integria IMS) we share all development planning tickets by releases with some of our partners, so that they can see in real time, the development planning based on “releases” and all the details of each ticket.

GitLab ticket view in Integria IMS/em>

Release ticket view

Development methodology used in Pandora FMS

At Pandora FMS, we have been using our own methodology from the beginning, although we have borrowed many ideas from agile methodologies, especially from SCRUM. From a life cycle point of view, we use an adaptation of the Rolling Release methodology

These are some important definitions when defining how we work, some of them come from Scrum, others from other methodologies.

Objectives of Pandora FMS work methodology

The objectives involve not only the development members, but also QA, the documentation team and part of the marketing team:

• • Maximum visualization: The entire team must see the same information, and it must flow from bottom to top and from top to bottom. By sharing objectives we will be able to do a more effective job.
  • What is not seen does not exist, which implies that all information relevant to the project must be reflected in the management, implemented with Gitlab. What is not seen does not exist, and what does not exist will not be taken into account for any purpose. Strictly following this methodology will allow everyone to be very aware of the planning:

-Strict deadline compliance.

-Advance planning without last minute modifications.

-Clearer information and in due time.

-Elimination of work peaks and etc.

• Integrity,, with an increasingly large and complex project, it is imperative to keep integrity during development. All code must follow standards..

Ticket

The ticket is the minimum work unit. There is a single person responsible for its completion and it is planned to be carried out in a milestone (version release).

A ticket is the way in which the development work is broken down, so a big feature will be made up by different tickets, on which ideally several people can work.

The ticket must contain a functional or description of the requirements, which can include diagrams, specifications, interface diagrams (mockup), test sets, examples, etc. In some cases it may even contain the analysis and design of the whole solution.

A completed ticket must perform as specified in the functional document (ticket) and the changes that have been made to these specifications must be reflected in the ticket.

The functional is key so that QA can validate a ticket or not. QA will have to reopen a ticket if it does not meet any of the functional aspects.

Members and working groups

Product Owner (PO)

The PO defines where Pandora FMS has to go, in contact with customers, support and
the “real” market situation, providing technical and functional guidelines but without getting involved in development as such.

Product Committee

Group of people who will meet permanently with the PO to agree where the product is going to, trying to ensure that all PO decisions are collegiate. It is made up of the leader of each Development, QA, Support, Projects and Documentation team.

Development Manager (DM)

The DM will manage the entire development cycle: define milestones, priorities, manage
individually all members and make operational decisions. The DM reports exclusively to PO and is the leader of the development team.

Development Team

They are in charge of the development of large features and product improvements, complete code refactoring, change development (small features), bug fixes and product maintenance improvements.

QA Team

They verify that each development atomic unit works as defined in the
specifications. They will also create and maintain an ecosystem of automated testing for both backend and user experience.

Support Team

They are the ones who deal directly with the client solving issues. Their experience with the product’s day-to-day means that their opinions must be taken into account, that is why they are part of the product committee.

Project team

They implement it on the end customer and are the ones closest to the customer, since they are often there before the project exists, and they usually offer ideas and all kinds of features in hand, for all purposes they are the “speaker” of the commercial department, therefore they are part of the product committee.

Training and Documentation Team

Responsible for training and the product’s documentation. They coordinate with the marketing team and the translation team.

Remote working

All team members (development, QA, documentation) telework freely. In fact, developers from Europe, Asia and America participate in Pandora FMS, and within Spain they are distributed throughout the national territory. We are a 100% distributed and decentralized company, although with traditional hierarchies.

In order to telework, we need each member to take responsibility for their work, be autonomous and commit to planning. Teleworking entails minimizing the need for oral communication and physical personal meeting, replacing them not with teleconferences, but with a precise use of the tools of the development process.

Development watch-keeping

A developer on the team is especially devoted to solving incidences involving code, in permanent connection with the support team (from 8 am to 8 pm, CEST). This allows not only to have maximum agility when solving a problem on a client, but also code changes are integrated into the code repository in an organized way.

Ticket creation and classification process

Any member of the company (including salespeople) can create a ticket in GitLab. This includes customers and partners, although in their case there is a prior filter by the support team and the sales team respectively.

The more detailed the ticket, the more unequivocal the development will be. Add images, gifs, animations and all the necessary clarifications. As well as the way to access the environment where the problem has been found or the contact persons. A developer will never contact a customer directly. If there is the need to interact with them, it will be done through the support or project team.

Nobody, except for the DM or PO, can change a ticket milestone. On creation, the ticket will not have an assigned milestone or assigned user. The task of defining which release a ticket belongs to is the responsibility of PO and DM exclusively.

When a ticket is finished and the developer thinks it should be reviewed by a colleague, they mention it in the merge request through @xxxxx. The review must be nominal. This review is independent of the code review carried out by the department manager.

General ticket workflow

• The ticket is assigned to a programmer by the DM. If it does not have a ticket assigned, the ticket will be auto-assigned. (See below the terms that regulate this system).
• The developer must understand/solve any questions that may arise after reading the functional document, if necessary, check with the DM or the author of the ticket. This must be done before starting to develop. Once read, you must, in order:

1. Evaluate (by assigning labels) its complexity and size, reaching a prior consensus with the DM.
2. Develop the feature following the ticket specifications
3. Document everything developed in the same ticket or, if required, in a new documentation ticket. This ticket must relate to the “parent” ticket by ticket #ID.
4. The developer must test its functionality at least in:
   -standard docker development environment
   -docker development environment with data.

• When it is deemed complete, it will be tagged ~ QA Pending and placed in the hands of QA.
• For each FEATURE ticket, there will be a reference person, generally from projects, support or even the PO itself. This person will be the one who will define part of the functional (together with the DM and PO), but above all, this person will be the reference person for the developer to ask any details during development, and most importantly, should see the development progress, step by step, so that it is validated.
• Any change to the functional will be reflected by the reference person in the ticket as comments, without altering the original functional.
• If there is a child documentation ticket, QA will validate the ticket using the documentation generated by the reference person, NOT by the functional of the ticket, validating the documentation and the feature at the same time.

Release planning

When creating a ticket, the milestone must be empty (not assigned) like the user. The only ones that can classify a ticket are: DM and PO.

A series of milestones have been defined to support the ticket classification process, some of them, those dated (releases), can be seen as milestones, while the rest should be seen as simple ticket containers.

• (Not allocated): It is the absence of milestones in a ticket. For all intents and purposes, this ticket “does not exist yet.” The DM and PO will validate each and every one of these tickets to see if they make sense in the product roadmap. No developer should take any of these tickets.
• Feature backlog: Tickets that will be made at some indeterminate time in the future that sooner or later will have to be addressed. No developer should take any of these tickets.
• Low priority bugs: Reported bugs with no priority assigned yet by PO/DM. No developer should take any of these tickets.
• STAGE: Tickets proposed by each department for planning in a product release. At each planning meeting, these tickets will be discussed, and moved to other milestones. At the end of the cycle start meeting, this milestone should be empty. The DM is the one who has the final decision as to which STAGE tickets are assigned to a certain release and which are not, relying on the product committee if necessary. No developer should take any of these tickets.
• XXX: Release XXX. Milestone that groups a series of tickets that will be released on a certain date. A milestone has a deadline associated with it. In the case of RRR releases, this date could change, in the case of LTS not.

1. The development of the tickets associated with a release must be finished 5 days before the scheduled day for the release. Tickets not completed before that date will be delayed to the next release and the delay will have to be justified to the DM.
2. There are two types of release milestones:
   -LTS: in April and November. They are 6 months apart.
   -Regular Releases (RRR): There will be 2 to 4 regular releases between LTS releases.

• A developer with no assigned tasks for a release, as long as there are no pending assignment tickets in the release milestones for the developer’s team, can take one of the unassigned tickets from:
  -The closest release, based on date.
  -Second closest release, based on date.

CICD

Pandora FMS developers integrate the code of their branches in a central repository several times a day, causing a series of automatic tests to be executed whose objective is to detect faults as soon as possible and improve the quality of the product.

These tests run dynamically in a series of executors or “runners”, some of them specific, for certain architectures (e.g., ARM), that execute static code analyzers, unit tests, and activate containers to carry out integration tests in a real installation of the application.

The generation of Pandora FMS packages is completely automated. Packages are generated every night from the development branch for manual testing. They can also be generated on demand by any developer or member of the QA or support teams, from any branch through the GitLab web interface.

When a release is made from the stable branch, in addition to package generation, a series of steps are executed that deploy them to Ártica’s internal package server, to SourceForge, to Ártica’s customer support environment, and that, likewise, update the Debian, SUSE and CentOS repositories along with the official Docker images.