Monitoring vs. Observability

“Monitoring” and “observability” are often used interchangeably, but these concepts have a few fundamental differences.

Monitoring is the process of using telemetry data to understand the health and performance of your application. Monitoring telemetry data is preconfigured, implying that the user has detailed information on their system’s possible failure scenarios and wants to detect them as soon as they happen.

In the classical approach to monitoring, we define a set of metrics, collect them from our software system, and react to any changes in the values of these metrics that are of interest to us.

For example:

Excessive CPU usage can indicate that we need to scale it up to compensate for increasing system loads;

A drop in successfully served requests after a fresh release can indicate that the newly released version of the API is malfunctioning;

Health checks process binary metrics that represent whether the system is alive at all or not.

Observability extends this approach. Observability is the ability to understand the state of the system by performing continuous real time analysis of the data it outputs.

Instead of just collecting and watching predefined metrics, we continuously collect different output signals. The most common types of signals – the three pillars of observability – are:

• Metrics: Numeric data aggregates representing software system performance;

• Logs: Time-stamped messages gathered by the software system and its components while working;

• Traces: Maps of the paths taken by requests as they move through the software system.

The development of complex distributed microservice architectures has led to complex failure scenarios that can be hard or even impossible to predict. Simple monitoring is not enough to catch them. Observability helps by improving our understanding of the internal state of the system.

Metrics

Choosing the right metrics to collect is key to establishing an observability layer for our software system. Here are a few different popular approaches that define a unified framework of must-have metrics in any software system.

USE

Originally described by Brendan Gregg, this approach focuses more on white-box monitoring – monitoring of the infrastructure itself. Here’s the framework:

• Utilization – resource utilization.

  • % of CPU / RAM / Network I/O being utilized.

• Saturation – how much remaining work hasn’t been processed yet.

  • CPU run queue length;

  • Storage wait queue length;

• Errors – errors per second

  • CPU cache miss;

  • Storage system fail events;

Note: Defining “saturation” in this approach can be a tricky task and may not be possible in specific cases.

Four Golden signals

Originally described in the Google SRE Handbook, the Four Golden signals framework is defined as follows:

• Latency – time to process requests;

• Traffic – requests per second;

• Errors – errors per second;

• Saturation – resource utilization.

RED

Originally described by Tom Wilkie, this approach focuses on black-box monitoring – monitoring the microservices themselves. This simplified subset of the Four Golden Signals uses the following framework:

• Rate – requests per second;

• Errors – errors per second;

• Duration – time to process requests.

Choosing and following one of these approaches allows you to unify your monitoring concept throughout the whole system and make it easier to understand what is happening. They complement one another, and your choice may depend on which part of a system we want to monitor. These approaches also don´t exclude additional business-related metrics that vary from one component of the software system to another.

Logs

System logs are a useful source of additional context when investigating what is going on inside a system. They are immutable, time-stamped text records that provide context to your metrics.

Logs should be kept in a unified structured format like JSON. Use additional log storage/visualization tools to simplify interaction with the massive amount of text data the software system provides. One very well-known and popular solution for log storage is ElasticSearch.

Traces

Traces help us better understand the request flow in our system by representing the full path any given request takes through a distributed software system. This is very helpful in identifying failing nodes and bottlenecks.

Traces themselves are hierarchical structures of spans, where each span is a structure representing the request and its context in every node in its path. Most common tracing visualization tools like Jaeger or Grafana display traces as waterfall diagrams showing the parent and child spans caused by the request.

Building an observable software system lets you identify failure scenarios and possible risks during the whole system life cycle. A combination of metrics, extensive log collection, and traces helps us understand what’s happening inside our system at any moment and speeds up investigations of abnormal behavior.

This article was just the first step. We’ve covered the standard approaches to metrics and briefly discussed traces and logs. But to implement an observable software system, we need to set up its components correctly to supply us with the signals we need. In part 2, we’ll discuss instrumentation approaches and modern standards in this field.

In the face of inflation and interest rate hikes, companies started tightening their belts by reducing staff to survive the uncertain times ahead. As of July, all this resulted in more than 400 companies laying off over 80 000 workers worldwide.

In this interview, we speak to Karina Dajoraitė-Dirvonskienė, Chief People Officer at Nord Security, who shares her insights into the mass layoff trend, its impact on the cybersec industry, and the situation at Nord Security. She also gives tips on how to bounce back after experiencing a layoff.

The current job market

What, in your opinion, are the main reasons behind current layoffs in the tech sector?

There are several factors. Some of them come from the global perspective, like the looming threat of recession with the inflation and interest rates shooting up as well as the changing behavior of consumers. Of course, no one can account for a global crisis. Still, once it happens, some action scenarios should be put in place to secure your organization and ensure it stays afloat without making drastic sacrifices.

Also, there could be other factors at play here, too – like overplanning your hiring quotas, too much reliance on outside funding, as well as, changing evaluation of companies with the focus of investors from growth to profit, and failure to raise additional capital as a result.

The situation at Nord Security

What is our current situation in Lithuania, Berlin, and Nord Security in general?

Nothing is changing at Nord Security. We continue growing and expanding our global team with new locations and hires. After more than a decade of being a profitable bootstrapped business, last quarter, we officially became a tech unicorn by raising $100 million from investors at a valuation of $1.6 billion. And for the future, that means more growth and more talent joining our community worldwide.

Do you believe that the current trend will affect hiring goals at Nord Security? If so, how?

I believe it will. We will use this global trend of layoffs as an opportunity to fill our vacant positions more quickly. As I’ve mentioned, the growth of our organization continues. Therefore we’re looking for new talent to join our team.

Why do you think the situation at Nord Security is different from that of tech companies, that are currently laying off large numbers of staff?

Actually, our co-founder, Tom Okman, has recently commented on the situation that Lithuanian businesses are mainly profitable or bootstrapped. That was the case with us, too. Therefore, there’s no need to rely on outside funding too much. Also, I think the cybersecurity market that we work in also has an impact here. It is constantly growing and is expected to reach $376.32 billion in 2029. Moreover, the pandemic, geopolitical unrest, and an increase in cyberattacks have highlighted the need for our products.

Cybersecurity threats will continue to exist as long as we have our online lives. As a result, not only the demand for cyber products and services is growing, but also the need for cybersecurity professionals. Currently, there are more than 36,000 open positions for cybersec experts in the public sector and more than 700,000 available jobs in the private sector globally. These factors are the main reasons why the cybersecurity industry hasn’t seen hiring freezes or layoffs on a broad scale.

What should be considered when planning recruiting volumes, and how does this help us maintain sustainable growth?

We plan our recruiting volumes meticulously with every team in the organization, evaluating the needs, pain points, and plans for growth. I think it’s important to have this alignment throughout the whole organization and with the key stakeholders if you want your recruitment volumes to be on track and ensure sustainability. So for our team, communication and alignment are key in this accord.

Bouncing back after a painful layoff

Can you give some advice to people who have experienced layoff? How can they cope with the stress of losing their job?

Well, first of all – not to panic. Of course, it’s never pleasant to lose your job. However, while some companies are overhired, others are experiencing shortages. Therefore, there’re still plenty of job opportunities available, especially for those with tech skills. This demand will continue to grow in the future, notwithstanding the current hardships. So here are some tips on what you can do to find your next employment easier:

• If your local market doesn’t have a suitable offer, check global companies – remote work is still very much a possibility.

• Be active on LinkedIn – if you have recruiters in your connections, don’t hesitate to drop them a line.

• Reach out to friends to refer you to their employer. Many companies have internal referral programs available. At Nord Security, we even launched an external referral program – you can check it HERE.

• Network – there are plenty of events and meetups to attend online or offline for techies. Who knows, maybe there’s another job opportunity waiting for you.