Pandora FMS presents you our Roadmap 2021 – 2023

In this article, we will introduce you to the new Pandora FMS Roadmap for the next 24 months (June 2021 – June 2023). For its creation, we had the participation of our clients and partners, who, through a survey, helped us choose all kinds of features and their priority.

It’s been really satisfying for us to complete this challenge, as it was one of those enthusiastically proposed among our closest goals.

• Warp update (Q2).
• Command center (Q2).
• New agent inventory report (Q2).
• Graphic agent installer for Mac (Q2).
• Services report (Q3).
• Policy auto-implementation (Q3).
• New visual console elements (Odometer, Simple graph) (Q3).
• Netflow: Data monitoring of the flows defined in the filter (Q3).
• Trend modules (Q3).
• Capacity planning modules (AI) (Q3).
• Enhanced anomaly detection (AI) (Q3).
• Authentication with KERBEROS (Q3).
• New service view widget for Dashboard (Q3).
• Basic network computer configuration management (Q4).
• Centralized agent update (Q4).
• APM (code application monitoring) (Q4).
• Security Center (Q4).
• IPv6 monitoring in SNMP with Satellite server (Q4).
• ITSM integration: SysAid, Zendesk, OTRS, Redmine, Jira, Zammad, TopDesk (Q4).
• Impact simulation in service view (2022+).
• Discovery: Google Cloud.
• AWS Monitoring improvements with Discovery: RDS for postgreSQL, Autoscaling groups, VPCS, Lampdas.
• Azure Monitoring improvements with Discovery: Databases, Storage, Data Factory, PostgreSQL, Event hubs.
• GIS Alerts (2022+).
• IPAM Report (2022+).
• Data consultation to agents in real time (2022+).
• Public/private certificate validation system in remote agent configuration (2022+).
• Load Balancing in API/Console (2022+).
• Automatic remote inventory with satellite (SNMP, WMI, SSH/Linux) (2022+).
• New view to show systems currently affected by a scheduled downtime (2022+).
• SNMP trap reports (top-N by source, type of trap, etc) (2022+).
• Desktop application to configure Pandora FMS agent and see its status (2022+).
• IOT on Satellite server (2022+).

Warp Update

A unified system that allows updating console, server and agents. Fully integrated into the console, which does everything with a single click without having to execute commands, copy files or pray for everything to go smoothly. Fast and centralized, in the case of deployment of centralized updates through the Metaconsole.

Command Center

Command Center is the long-awaited evolution of the Metaconsole, which will allow dozens of nodes to be managed in a totally transparent and centralized way simultaneously, without having to manually synchronize any element.

Security Center

An innovative way to manage server and workstation security, fully integrated with system monitoring.

APM in source code

We want to reach the last frontier of monitoring, the code in applications to measure their times and detect bottlenecks and overloads, combining all the information on the same platform where the infrastructure, servers and application metrics are.

Trend modules

Create a new type of predictive module that compares two time ranges and evaluates, in a percentage or an absolute way, their differences. These modules can be used in alerts, graphs or reports.

E.g.: Access router outbound traffic is 25% higher than last month. This month there are 22 new users compared to the previous month.

Centralized agent update

Update agents centrally from the console. A current enhancement to the remote agent distribution system.

Network computer configuration management

Being able to edit “download” and “upload” full configurations of network equipment through several protocols (TFTP, Telnet, SSH) in order to centrally manage network equipment such as switches and routers. Some of its purposes:

• Schedule configuration backups, restore trusted configuration versions with a single click.
• Detect changes in real time and know “who”, “what” and “when” about configuration changes.
• Upgrading device firmware.
• Save time by automating time-consuming and repetitive tasks using templates and configuration application scripts.
• Make sure changes made to running configurations are saved.
• Compare NVRAM (running) configurations with startup ones (saved) to identify changes that need to be saved.
• Quickly identify and correct unauthorized or failed changes (restoring backup manually).
• Compare configurations with base configurations to identify and reverse unwanted changes.

Netflow

Be able to integrate simple data from a Netflow filter as a Pandora FMS numerical module, to be able to, for example, set alarms when the traffic of a certain flow exceeds its threshold or to be able to measure SLA in flow traffic.

Capacity planning modules

Modules that operate like Capacity Planning reports and can estimate in a future time threshold, e.g.: 1 month, 3 months, the value of a given module, estimating its growth based on a statistical analysis of its history.

Policy auto-implementation

Add a policy self-enforcement system (optional) that works well. Either based on the detection of new elements (in the added group or directly in agents in the policy itself) or even just the possibility of scheduling policy application at a certain time interval.

Data consultation to agents in real time

Upon manual request: configuration data, status, hardware status, OS items, logs, etc., in real time, all from a library of predefined elements. Without complementary configuration, it would only need the deployment of an additional agent to that of Pandora FMS. These data are only for screen display, not for making alerts or reports. The data range would be very broad and standard. It requires direct connectivity from the console and the agent that must listen on a specific port.

Service report

Reports to show service SLA compliance, numerically (%) and with a histogram.

IPAM reports

New reports to, among other things, show the usage percentage of each network, and some other information of interest that appears on the IPAM screens but that cannot be included in reports.

GIS alerts

Be able to send alerts when an agent leaves a delimited coordinate zone, which is often called “geo-fencing”.

Load balancing in Console and API

Provide a standard system that allows load balancing in the console and the API, in order to scale and distribute the load. Perfect for environments where the use of the API is intensive or the console is used in multi tenant environments.

IoT

Offer support to the Satellite Server to natively support modbus and MQTT protocols.

It’s been hard work, but thanks to Pandora FMS employees and our partners and clients, we achieved this Roadmap 2021 – 2023 that will make our work easier in the future and speed it up.

SaaS vs onPremise, do you use the cloud?

Do you use the cloud?

Be aware that we’re not saying that you are in cloud nine, but that you may most likely be using the cloud. That is, if you use Google mail, Microsoft Office 365 office suite or you take a photo with your cell phone and then it gets automatically uploaded to iCloud or something similar, you are using the cloud.

The cloud, as an abstract concept, encompasses a series of technical terminology such as SaaS, IaaS, PaaS, etc. The good thing about the concept of the cloud is that you can guess what it does thanks to the metaphor: we do not know where our data are, or how they get there, nor does it matter much for us, because it is far away and it does not affect us. The great success of the cloud of the 21st century has been to find an especially powerful metaphor that omits the complexity behind that technology and gives us peace of mind.

The concept of using third-party infrastructure for “our stuff” is the oldest thing in computing. In fact, back in the 60s of the last century, most computing worked like this. You connected to a large machine from a computer that was not as such, but a screen and a keyboard. Then the microcomputer craze turned around and every computer was self-sufficient. Now, almost a century later, we have rediscovered that it is more efficient to have everything centralized in one big system.

I have nothing against the cloud. Well, my life is not at stake, unless for example, I entrust the IT infrastructure of my business to the cloud. This is what happened to a number of companies in Asia, such as CITEX or BitMax that used the Amazon cloud (AWS) to host their Bitcoin exchange service (Exchangers), well, them and also the Asian sites from Adobe, Business Insider, Expedia, Expensify, FanDuel, FiftyThree, Flipboard, Lonely Planet, Mailchimp, Medium, Quora, Razer, Signal, Slack, Airbnb, Pinterest, SendGrid and a few hundred more. The cloud is not infallible, the cloud is comfortable.

Today many companies have relied so much on the cloud that it is impossible to take a step back, get out of the cloud, because they would literally have to remake the system with another technology. The cloud is easy but implies total dependence on the provider, especially in technologically optimized systems such as Amazon’s. It’s too good a candy to resist.

Realistically, if you’ve already risen to the sky and are floating with the clouds, and the technology that supports your business is floating above your head, it may not be easy or comfortable to go back, in fact, you may have probably already realized that the cloud is not cheap at all and the costs are increasing over time, and are difficult to predict.

Well, it’s already in, and it’s not going to change, so you should at least be able to keep an eye on what your provider is doing. Monitor the quality of service they offer you and make sure for yourself, because who is watching the watchdog? That’s right, do it yourself, trust no one, do it with your own systems, don’t use a cloud system to monitor another cloud system, put your feet on the ground and buy yourself an umbrella, just in case it rains.

The “lifetime” model: onPremise

On the contrary, we have the classic model of “buying the software” and using it however you want, wherever you want and, whenever you want you change programs without much thought. Oddly enough, this is really the new model, the pay-per-use model that SaaS has copied predates conventional software licenses. The onPremise model gives you the right to use the software on your own computers, in your own facilities and where the manufacturer or software owner does not have any access or rights. The only requirement is to pay for it and use it under the conditions approved by the license you acquired.

Cost analysis: onPremise vs SaaS

The onPremise model has some undeniable advantages, the main one being data security. As it is running on your systems, you own both the information and the processes that use that information. This has legal and business implications, since changing providers can be easier than when you use its SaaS equivalent.

Although it may seem a lie, in the long term the SaaS model is more expensive than the onPremise model, and above all, with the onPremise model it is much easier to estimate the Total Cost of Ownership (TCO) in the medium term. This can be easily demonstrated if we compare the costs in the subscription/pay-per-use model (SaaS) and the license ownership model (onPremise) for one, three and five years.

• Suppose a SaaS license annual cost is €5,000/year. In this case it is pure OPEX (operating costs).
• Let’s picture an onPremise license whose annual cost is €10,000 the first year, and whose annual maintenance cost is 20% (which is the standard in the market). That supposes a renewal cost of €2000/year. In this case, it is pure CAPEX (investment in assets, software).

There are intangible factors, such as input barriers, higher in onPremise models, and output barriers, higher in SaaS models. It is also true that an onPremise installation involves additional costs: those of infrastructure, operation and training.

In certain types of applications with little added value such as office tools, the SaaS model is here to stay. Office 365 or Google Docs are a perfect example.

In other cases, such as Adobe Photoshop, the onPremise model has been combined with a pay-per-use -subscription- model (but without being SaaS) combined with the conventional onPremise licensing model.

Summary of arguments in favor of each model

A few rules for safe password management

In this, our competent blog, we boast of always giving you good advice and providing you with the technological information necessary for your life as a technologist to make sense. Today it is the case again, we will not reveal the hidden secret about the omnipotence of Control/Alt/Delete, but almost. Today in Pandora FMS blog, we give you a few tips for safe password management.

Safe password management

The purpose of this article is for users to be responsible for keeping their coveted passwords or authentication information safe when accessing confidential information. Because think about it, dear reader, how long ago did you come up with your first password? Surely it was to enter your select club in the treehouse. Maybe you even still choose the same for your social networks, Netflix or office pc. Was it as ordinary as your birth date? Your name and the first two acronyms of your surname? “RockyIV”, which was the name of your fourth favorite pet and movie? I don’t blame you, we have all been equally original and carefree when choosing a password.

But that is over! Many things already depend on this password, on this motto or pass that must include more than eight characters and at least one capital letter and one number. Your company security is not a game, damn it! There is a lot of mischief and felon out there that can put you and your businesses in a loophole, because of a vulnerability such as having a poor password! But do not worry, we will help you, we will talk about safe password management. We are Pandora FMS blog, we like potato salad, Kubrick movies and fighting against injustices!

Recommendations for safe password management

*Obvious but vital fact: User IDs and passwords are used to check the identity of a user on systems and devices. I just point that out here as an outline in case someone is so lost that they don’t know this. I repeat that we are talking about strong password management, so knowing what a password is is a must and saves time.

Said passwords are necessary for users to have access to information, normally, even if the merit is not recognized: capital information in your company. User IDs and passwords also help ensure that users are held accountable for their activities on the systems they have access to. Because yes, telereader friend, users are responsible for any activity associated with their user IDs and passwords. For that reason, it is very important for you to protect the password with your life and comply with the following policies related to them:

1. Users may not, under any circumstances, give their password or a password indication to a third party. *This seems obvious, but trust me, it is not. People sneak passwords like they’re office whispers or reggaeton choruses.
2.  Users will not use user identifiers or passwords of other users. *As we can see, in this case, sharing is not living.
3.  Users must change initial passwords or passwords received as temporary “reset” passwords immediately upon receipt. *For me, this is the most exciting and creative part, you never want to set the abstract code they give you, you want to improvise, imagine, CREATE!
4.  Users should change their passwords if they suspect that their confidentiality may have been compromised, and immediately report the situation as a security incident. *Don’t be ashamed of yourself, admit that someone may have violated your secret and repent before it’s too late.
5.  Users should not use the “remember password” function of programs. For example, if an application sends users the message of “automatically remember or store” the user’s password for future use, they will have to reject it. *This is a piece of information you did not know, huh? Well, it is as interesting as it is important.
6.  Users should not store passwords without encryption, for example, in a text file or an office document. In this case, this document must be protected with access control.
7.  When an administration password must be communicated, never send by the same means, the user and the password. For example, the user should be sent by email and the password by instant messaging. *I know that sometimes you try to save time, but with these things you better take your time and do not risk it.
8.  Users should not set the password on a post-it on the monitor, nor on the table, nor in the drawer or “hidden” in another place in the office or among your personal belongings. *This is one of the big mistakes everyone makes. Yes, post-its or notebook sheets have always helped us, but this time they are too obvious to keep such a big secret.
9.  Users should not use the same password for two systems or different applications. *Sorry, but you will have to memorize more than one. But rest assured, if a chimpanzee could recognize the descending sequence of nine numbers, someone who graduated from elementary school can do better.
10.  Users who find out the password of other users must report it, ensuring it is changed as soon as possible. *Here fellowship first and foremost. It is not only right hugging after company dinners. Camaraderie above all!
11.  Users must change their passwords at least once a year, or when indicated by the system, and in the case of administration passwords every 180 days, or in the event of changes of personnel in the company that may know them.
12.  If now you are afraid because you do not have a strong enough password, it’s normal, but I repeat, calm down, follow the following rules for passwords creation (if the system supports them) and nothing will go wrong:

•  a) Passwords must be at least six characters long.
•  b) Passwords must not be easily predictable and must not be contained in dictionaries. For example: your username, date of birth, or 1234, we all know that one.
•  c) Passwords must not contain consecutive repeating characters. For example: “AABBCC”.
•  d) Passwords must have at least an alphanumeric character, a numeric character, and a special character.

Good, and so far that was the lecture about being responsible that you must assume and internalize if you want things to go smooth at least in terms of passwords and vulnerabilities. Oh, nothing to thank us for! You know: “Life is beautiful. Password yourself”. Look, that could be your new password, right? No, the answer is NO! REMEMBER EVERYTHING WE LEARNED TODAY IN THIS ARTICLE!

Introduction

Do an exercise, ask five IT technicians -of any profile- what SNMP means.. If you’re close with them the better, so that the first thing they do is not go to Wikipedia to boast. Hopefully, they might tell you what they said to me when I was working in networks.

“Security is Not My Problem”

Taking into account that the SNMP protocol is one of the monitoring bases, and a system that has been in use for more than thirty years, this answer, “Security is Not My Problem”, sums up the current monitoring situation quite well: ignorance, laziness and lack of interest in monitoring security.

By the way, we talked about SNMP in another article on our blog and I will give you a teaser in advance, it means Simple Network Management Protocol and it comes from 1987.

Considering that monitoring is “key to the kingdom”, since it allows access to all systems and even access many times with administration credentials, shouldn’t we take security a little more seriously when we talk about it?

Recent vulnerabilities in well-known monitoring systems such as Solarwinds or Centreon make the need to take security seriously in the implementation of monitoring systems increasingly urgent, since these have a very strong integration with systems.

In many cases, security problems are not so much about one piece of software being much safer than another, but about poor configuration and/or architecture. It must be taken into account that a monitoring system is complex, extensive, and in general, it is highly adapted to each organization. Today it was Solarwinds, tomorrow it could be Pandora FMS or Nagios.

No application is 100% secure, nor is any corporate network secured against intrusion, whatever the type. This is an increasingly evident fact and the only thing that can be done about it is to know the risks and assume which ones you can take, which ones absolutely not, and work on the latter.

Safe monitoring architecture

It is essential to keep in mind at all times that a monitoring system contains key information for a possible intruder. If monitoring falls into the wrong hands, your system will be compromised. That is why it is so important to devote time to the architecture of your monitoring system, whatever it may be.

Carry out a first analysis, collecting the requirements and scope of your monitoring strategy:

• Identify what systems you are going to monitor and catalogue their security levels.
• Identify which profiles will have access to the monitoring system.
• Identify how you will obtain information from those systems, whether through probes/agents or remote data.
• Identify who is responsible for the systems you are going to monitor.

The architecture of a system will have, whatever the chosen software, the following elements and will have to take into account its network topology, its resources and the way to protect them properly:

1. Information display interface (web console, heavy application).
2. Data storage (usually a relational database).
3. Information collectors (intermediate servers, pollers, collectors, etc.).
4. Agents (optional).
5. Notification system (alerts, notices, etc.).

Monitoring system securing

No matter how correct the implementation of a system, its architecture and its design as a whole is, if one of the elements that make it up is violated, the damage it may suffer by a malicious attack compromises the entire structure. For this reason, in security there is a saying, “Security is a chain and your real security always depends on its weakest point.”

This list of security concepts applied to the architecture of a monitoring system can be summarized as the features that a monitoring product must have to ensure maximum security in an implementation:

• Encrypted traffic between all its components.
• High availability of all its components.
• Integrated backup.
• Double access authentication.
• Delegated authentication system (LDAP, AD, SAML, Kerberos, etc.).
• ACL and user profiling.
• Internal audit.
• Password policy.
• Sensitive data encryption.
• Credential containers.
• Monitoring of restricted areas/indirect access.
• Installation without superuser.
• Safe agent/server architecture (passive).
• Centralized and distributed update system.
• 24/7 support.
• Clear vulnerability management policy by the manufacturer.

Monitoring infrastructure basic securing

The management console, monitoring servers and other elements should never be on an accessible public network. The console should always be protected on an internal network, protected by firewalls and, if possible, on a network independent from other management systems.

The operating systems that host the monitoring infrastructure should not be used for other purposes: for example, to reuse the database for other applications, nor the base operating systems to run other applications.

Safe and encrypted traffic

You should make sure that your system supports SSL/TLS encryption and certificates at both ends at all levels: user operation, communication between components or sending data from the agent to the servers.

If you are going to use agents in unsafe locations, it is highly recommended that you force all external agents to use certificate-based authentication at both ends, to avoid receiving information from unauthorized sources and to prevent information collected by agents to not travel transparently.

On the other hand, it is very important for you to activate encryption on your web server to provide an encrypted administration console and prevent any attacker from seeing access credentials, remote system passwords or confidential information.

Full High Availability

For all elements: database, servers, agents and console.

Integrated backup

The tool itself should make this as easy as possible, as settings and data are often highly distributed and consistent backup is complex.

Clear vulnerability management policy by the manufacturer

Every day, dozens of independent auditors test the strengths and weaknesses of all kinds of business applications. They seek to gain a foothold in the sector by publishing an unknown ruling to increase their reputation. Many clients, as part of their internal security management processes, execute external and internal security audits that target their IT infrastructure.

Be that as it may, all products have security flaws, the question is: how are those flaws handled? Transparency, diligence and communication are essential to prevent customers from having problems derived from vulnerabilities in the software they use. It is essential that there is a clear policy in this regard, so that it is known which public vulnerabilities have been reported, when they have been corrected and if a new one is detected, the steps to follow for notification, mitigation and distribution to the end customer.

Dual authentication system

Pandora FMS has an -optional- system based on google authenticator that allows forcing its use for all users for security policies. This will make user access to the administration console much safer, preventing that due to privilege escalation the system can be accessed as administrator, which is, at best, the highest risk that can be run.

Delegated authentication system

Complementary to the previous one, you can delegate management console authentication to authenticate against LDAP, Active Directory, or SAML. It will enable a centralized access management, and combined with the double authentication system your access will become much safer.

ACL and user profiling

Identify and assign different users to specific people. Do not use generic users, assign only the necessary permissions and do not use “super administrators”. They are good practices not only for monitoring tools but for any business software implementation with access to sensitive information.

Nowadays, any professional tool to define an access profile for each user will do so in such a way that no user has “absolute control”, but only has the minimum required access to their functions.

Internal audit system

You must have a system in place to record all user actions, including information on altered or deleted fields. Said system must be able to be exported abroad so that not even the administrator user can alter said records.

Password policy

A basic element that allows you to enforce a strict password management policy for access to application users: minimum password size, password type, their reuse, forced change once in a while, etc.

Sensitive data encryption

The system must allow the most sensitive data to be stored encrypted and safely, such as access credentials, monitoring element custom fields, etc. Even if the system itself contains the encryption “seed”, it will always be much more difficult for a potential attacker to access this information.

Credential containers

Or an equivalent system for the administrator to delegate credential use to other users who use said credentials to monitor elements without seeing the passwords contained in the container.

Restricted area monitoring

In these systems, information will be collected remotely by a satellite server and will be available to be collected from the central system (in Pandora FMS through a specific component called Sync server). That way, data can be collected from a network without access to the outside, ideal for very restrictive environments where the impact is drastically reduced if an attacker takes over the system.

Agent remote management locking system

For critical security environments, where the agent cannot be remotely managed once it is configured. This is especially critical in monitoring, since if a system is compromised and its administration is accessed, by the way the system is configured itself, it will have access to all systems from where it receives information. In critical systems, the remote management capacity must be deactivated, even if that makes administration more tricky. The same applies to automatic updates on the agent.

Design of safe architecture for communication with agents

Sometimes known as passive communication. That way, agents will not listen to a port nor have remote access from the console. They are the ones who will connect to the central system to ask for instructions.

Installation without root

Pandora FMS can be installed in environments with custom paths without running with root. In some banking environments, it is a requirement that we meet.

Notification and reporting system (alerts, notices, etc.)

A monitoring system is only useful if it shows accurate information when it is needed. Alert or weekly report reception is the culmination of all the previous work and for that you will have to take into account some “obvious” points that are often overlooked. Protect those systems, wherever they may be.

Periodic updates

All manufacturers now distribute regular updates, which include both bug fixes and security problems. In our case, we publish updates approximately every five weeks. It is essential to update systems as soon as possible, because when a vulnerability is reported, product managers ask external security researchers who have reported the bug, not to publish anything about the vulnerability until a patch is published. Once the patch is published, the researcher will publish the information in more detail as wished, a fact that can be used to exploit and attack non-updated software versions.

Pandora FMS has a vulnerability disclosure public policy as well as a public catalog of known and reported vulnerabilities. Our policy has maximum transparency and full communication with security researchers, always to mitigate the impact of any security problem and to be able to protect our clients as a top priority.

24/7 support

In our support, the technician who answers the phone has the whole team backing him up. If there is a security issue and a security patch has to be published within hours. We not only have the technology to spread the patch to all our customers, but also the team to develop it in record time.

Base system securing

Hardening or system securing is a key point in the global security strategy of a company. As manufacturers, we issue a series of recommendations to carry out a safe installation of all Pandora FMS components, based on a standard RHEL7 platform or its equivalent Centos7. These same recommendations are valid for any other monitoring system:

Hardening checklist for monitoring base system:

• System access credentials.
• Superuser access management.
• System access audit.
• SSH securing.
• Web server securing.
• DB server securing.
• Server minimization.
• Local monitoring.

Access credentials

To access the system, nominative access users will be created, without privileges and with access restricted to their needs. Ideally, the authentication of each user should be integrated with a double authentication system, based on token. There are free and safe alternatives such as Google Authenticator that can be easily integrated into Linux, although outside the scope of this guide. Seriously consider its use.

If it is necessary to create other users for applications, they must be users without remote access (for this, it is necessary to deactivate their Shell or some equivalent method).

Superuser access through sudo

In the event that certain users must have administrator permissions, SUDO will be used.

Base system access audit

It is necessary to have the security log /var/log/secure active and monitor those logs with monitoring (which we will see later).

By default CentOS has this enabled. If not, just check the /etc/rsyslog.conf or /etc/syslog.conf file.

We recommend you to take the logs from the audit system and collect them with an external log management system. Pandora FMS can do it easily and it will be useful to set alerts or review them centrally in case of need.

SSH server securing

The SSH server allows you to remotely connect to your Linux systems to execute commands, so it is a critical point and must be secured by paying attention to the following points:

• Modify default port.
• Disable root login.
• Disable port forwarding.
• Disable tunneling.
• Remove SSH keys for remote root access.
• Investigate the source of keys for remote access. To do this, look at the content of the file /home/xxxx/.ssh/authorized_keys and see which machines they are from. Delete them if you think there shouldn’t be any.
• Establish a standard remote access banner that clearly explains that the server is a private access server and that anyone without credentials should log out.

MySQL server securing

Listening port. If MySQL server has to provide service to the outside, just check that the root credentials are safe. If MySQL only gives service to an internal element, make sure that it only listens on localhost.

Web server securing

We will modify the configuration to hide the Apache and OS version in the server information headers.

If you use SSL, disable unsafe methods. We recommend the use of TLS 1.3 only.

System service minimizing

This technique can be very exhaustive. It consists simply of eliminating everything that is not necessary in the system. Thus we avoid possible problems in the future with poorly configured applications that we really did not need and that can be vulnerable in the future.

Local monitoring

All the internal monitoring systems would have to be monitored to the highest level, specially information registries. In our case the following active controls in addition to the standard controls are always recommended:

• Active security Plugin.
• Complete system inventory (specially users and installed packages).
• System logs and server security.

Pandora FMS started as a totally personal open source project back in 2004. I wasn’t even a professional programmer, I was doing Unix security consulting. In fact, I chose PHP but Pandora FMS was my first application with PHP, I knew some things about ASP and my favorite programming language had been C.

A project with a single programmer and no professional users of his software yet is very different from a project with several dozen programmers and hundreds of clients using the software in critical environments. The evolution that Pandora FMS has undergone from 2004 to 2021 is a real case of steady improvement in software engineering.

Fortunately, I did not pay much attention to that subject of the degree, because most of the things that work and that I have learned with practice do not come in a book, nor are they explained at the university, because each software project and each team of people is very different. It may sound cliché, but it is the truth, and it is better to accept it and avoid formulas, because building a solid software product that can grow over time is not trivial at all.

In this article, I am going to talk about our experience, our evolution over time, but above all, about how our engineering processes work today. I have always believed that the most important part of open source is transparency, and that this should apply to everything, not only to software but also to processes and knowledge in general.

Version control system

It is an essential part of any software project. Today the ubiquitous GIT is everywhere (by the way, not everyone knows that Git is the work of Linus Torvalds, original author of the Linux kernel). A version control system helps, in short, a group of developers work without overlapping their jobs.

When the Pandora FMS project started, I was working without version control, because there were no other people. When some people began to collaborate on it, we realized that a simple shared directory was not worth it, because we were overlapping the code and, yes, making backups to save old versions was not a very efficient method.

The first version control system we used was CVS, which we have been using for eight years or more. Around 2008, we started using SVN (Subversion) another slightly more efficient system and it wasn’t until 2013 when we started using GIT and opened our official repository on Github.

Pandora FMS public repository on Github

Since Pandora FMS has an open source version and an Enterprise version -with proprietary code and commercial licenses- we have two GIT projects, one public on GitHub and the other private, which we manage with GitLab. The GitHub version is in sync with our private copy on GitLab at our offices. Some partners who collaborate with us in developing have access to this private repository, and through an extension of our support application (Integria IMS) we share all development planning tickets by releases with some of our partners, so that they can see in real time, the development planning based on “releases” and all the details of each ticket.

GitLab ticket view in Integria IMS/em>

Release ticket view

Development methodology used in Pandora FMS

At Pandora FMS, we have been using our own methodology from the beginning, although we have borrowed many ideas from agile methodologies, especially from SCRUM. From a life cycle point of view, we use an adaptation of the Rolling Release methodology

These are some important definitions when defining how we work, some of them come from Scrum, others from other methodologies.

Objectives of Pandora FMS work methodology

The objectives involve not only the development members, but also QA, the documentation team and part of the marketing team:

• • Maximum visualization: The entire team must see the same information, and it must flow from bottom to top and from top to bottom. By sharing objectives we will be able to do a more effective job.
  • What is not seen does not exist, which implies that all information relevant to the project must be reflected in the management, implemented with Gitlab. What is not seen does not exist, and what does not exist will not be taken into account for any purpose. Strictly following this methodology will allow everyone to be very aware of the planning:

-Strict deadline compliance.

-Advance planning without last minute modifications.

-Clearer information and in due time.

-Elimination of work peaks and etc.

• Integrity,, with an increasingly large and complex project, it is imperative to keep integrity during development. All code must follow standards..

Ticket

The ticket is the minimum work unit. There is a single person responsible for its completion and it is planned to be carried out in a milestone (version release).

A ticket is the way in which the development work is broken down, so a big feature will be made up by different tickets, on which ideally several people can work.

The ticket must contain a functional or description of the requirements, which can include diagrams, specifications, interface diagrams (mockup), test sets, examples, etc. In some cases it may even contain the analysis and design of the whole solution.

A completed ticket must perform as specified in the functional document (ticket) and the changes that have been made to these specifications must be reflected in the ticket.

The functional is key so that QA can validate a ticket or not. QA will have to reopen a ticket if it does not meet any of the functional aspects.

Members and working groups

Product Owner (PO)

The PO defines where Pandora FMS has to go, in contact with customers, support and
the “real” market situation, providing technical and functional guidelines but without getting involved in development as such.

Product Committee

Group of people who will meet permanently with the PO to agree where the product is going to, trying to ensure that all PO decisions are collegiate. It is made up of the leader of each Development, QA, Support, Projects and Documentation team.

Development Manager (DM)

The DM will manage the entire development cycle: define milestones, priorities, manage
individually all members and make operational decisions. The DM reports exclusively to PO and is the leader of the development team.

Development Team

They are in charge of the development of large features and product improvements, complete code refactoring, change development (small features), bug fixes and product maintenance improvements.

QA Team

They verify that each development atomic unit works as defined in the
specifications. They will also create and maintain an ecosystem of automated testing for both backend and user experience.

Support Team

They are the ones who deal directly with the client solving issues. Their experience with the product’s day-to-day means that their opinions must be taken into account, that is why they are part of the product committee.

Project team

They implement it on the end customer and are the ones closest to the customer, since they are often there before the project exists, and they usually offer ideas and all kinds of features in hand, for all purposes they are the “speaker” of the commercial department, therefore they are part of the product committee.

Training and Documentation Team

Responsible for training and the product’s documentation. They coordinate with the marketing team and the translation team.

Remote working

All team members (development, QA, documentation) telework freely. In fact, developers from Europe, Asia and America participate in Pandora FMS, and within Spain they are distributed throughout the national territory. We are a 100% distributed and decentralized company, although with traditional hierarchies.

In order to telework, we need each member to take responsibility for their work, be autonomous and commit to planning. Teleworking entails minimizing the need for oral communication and physical personal meeting, replacing them not with teleconferences, but with a precise use of the tools of the development process.

Development watch-keeping

A developer on the team is especially devoted to solving incidences involving code, in permanent connection with the support team (from 8 am to 8 pm, CEST). This allows not only to have maximum agility when solving a problem on a client, but also code changes are integrated into the code repository in an organized way.

Ticket creation and classification process

Any member of the company (including salespeople) can create a ticket in GitLab. This includes customers and partners, although in their case there is a prior filter by the support team and the sales team respectively.

The more detailed the ticket, the more unequivocal the development will be. Add images, gifs, animations and all the necessary clarifications. As well as the way to access the environment where the problem has been found or the contact persons. A developer will never contact a customer directly. If there is the need to interact with them, it will be done through the support or project team.

Nobody, except for the DM or PO, can change a ticket milestone. On creation, the ticket will not have an assigned milestone or assigned user. The task of defining which release a ticket belongs to is the responsibility of PO and DM exclusively.

When a ticket is finished and the developer thinks it should be reviewed by a colleague, they mention it in the merge request through @xxxxx. The review must be nominal. This review is independent of the code review carried out by the department manager.

General ticket workflow

• The ticket is assigned to a programmer by the DM. If it does not have a ticket assigned, the ticket will be auto-assigned. (See below the terms that regulate this system).
• The developer must understand/solve any questions that may arise after reading the functional document, if necessary, check with the DM or the author of the ticket. This must be done before starting to develop. Once read, you must, in order:

1. Evaluate (by assigning labels) its complexity and size, reaching a prior consensus with the DM.
2. Develop the feature following the ticket specifications
3. Document everything developed in the same ticket or, if required, in a new documentation ticket. This ticket must relate to the “parent” ticket by ticket #ID.
4. The developer must test its functionality at least in:
   -standard docker development environment
   -docker development environment with data.

• When it is deemed complete, it will be tagged ~ QA Pending and placed in the hands of QA.
• For each FEATURE ticket, there will be a reference person, generally from projects, support or even the PO itself. This person will be the one who will define part of the functional (together with the DM and PO), but above all, this person will be the reference person for the developer to ask any details during development, and most importantly, should see the development progress, step by step, so that it is validated.
• Any change to the functional will be reflected by the reference person in the ticket as comments, without altering the original functional.
• If there is a child documentation ticket, QA will validate the ticket using the documentation generated by the reference person, NOT by the functional of the ticket, validating the documentation and the feature at the same time.

Release planning

When creating a ticket, the milestone must be empty (not assigned) like the user. The only ones that can classify a ticket are: DM and PO.

A series of milestones have been defined to support the ticket classification process, some of them, those dated (releases), can be seen as milestones, while the rest should be seen as simple ticket containers.

• (Not allocated): It is the absence of milestones in a ticket. For all intents and purposes, this ticket “does not exist yet.” The DM and PO will validate each and every one of these tickets to see if they make sense in the product roadmap. No developer should take any of these tickets.
• Feature backlog: Tickets that will be made at some indeterminate time in the future that sooner or later will have to be addressed. No developer should take any of these tickets.
• Low priority bugs: Reported bugs with no priority assigned yet by PO/DM. No developer should take any of these tickets.
• STAGE: Tickets proposed by each department for planning in a product release. At each planning meeting, these tickets will be discussed, and moved to other milestones. At the end of the cycle start meeting, this milestone should be empty. The DM is the one who has the final decision as to which STAGE tickets are assigned to a certain release and which are not, relying on the product committee if necessary. No developer should take any of these tickets.
• XXX: Release XXX. Milestone that groups a series of tickets that will be released on a certain date. A milestone has a deadline associated with it. In the case of RRR releases, this date could change, in the case of LTS not.

1. The development of the tickets associated with a release must be finished 5 days before the scheduled day for the release. Tickets not completed before that date will be delayed to the next release and the delay will have to be justified to the DM.
2. There are two types of release milestones:
   -LTS: in April and November. They are 6 months apart.
   -Regular Releases (RRR): There will be 2 to 4 regular releases between LTS releases.

• A developer with no assigned tasks for a release, as long as there are no pending assignment tickets in the release milestones for the developer’s team, can take one of the unassigned tickets from:
  -The closest release, based on date.
  -Second closest release, based on date.

CICD

Pandora FMS developers integrate the code of their branches in a central repository several times a day, causing a series of automatic tests to be executed whose objective is to detect faults as soon as possible and improve the quality of the product.

These tests run dynamically in a series of executors or “runners”, some of them specific, for certain architectures (e.g., ARM), that execute static code analyzers, unit tests, and activate containers to carry out integration tests in a real installation of the application.

The generation of Pandora FMS packages is completely automated. Packages are generated every night from the development branch for manual testing. They can also be generated on demand by any developer or member of the QA or support teams, from any branch through the GitLab web interface.

When a release is made from the stable branch, in addition to package generation, a series of steps are executed that deploy them to Ártica’s internal package server, to SourceForge, to Ártica’s customer support environment, and that, likewise, update the Debian, SUSE and CentOS repositories along with the official Docker images.