Skip to content

Cloud-Native TACACS+: Modern Network Device Administration

What is TACACS+?

TACACS+ is a remote authentication protocol that allows a remote access server to communicate with an authentication server in order to validate a user’s access to a network device such as a wireless access point or wired switch (i.e. network device administration).

Cloud-Native TACACS+ by Portnox

Portnox TACACS+-as-a-Service is the first and only cloud-native solution for network device administration – authentication, authorization, and accounting (AAA). TACACS+ by Portnox enables organizations to maintain transparent and secure administration of network devices by centralizing user authentication, access control policy enforcement, activity audit trails, and more – all from the cloud.

Authentication for Network Devices

Strengthen network device administration and improve organizational efficiency by authenticating users via Open LDAP, or Active Directory integration. Portnox supports Azure AD, Google Workspace, Microsoft AD, and OKTA.

Authorization & Access Control

Easily enforce network device access control policies that limit configuration changes to maintain administrative security. Dictate privilege levels, allowed services, the use of specific autocommands, custom attributes, and more.

Audit Trails & Accounting

Track user activity and attributes across network devices such as identities, start and stop times, executed commands, packet transfers, and much more to help maintain administrative transparency and streamline security audits.

Key TACACS+ Features from Portnox

System ArchitectureAuthentication MethodsAuthorization PoliciesAccounting RecordsIntegrations
Cloud-nativeAzure ADPrivilege levelsUser identitiesSIEM
MS Azure-hostedGoogle WorkspaceSession timeout valueStart / stop timesActive Directory
No upgradesMicrosoft ADAutocommandsExecuted commandsRESTful API
No patchesOKTAAllowed servicesPacket transfers
Hardware agnosticOpenLDAPCustom attributesUS-ACII strings
…and more

Network Device Administration with Portnox

About Version 2 Digital

Version 2 Digital is one of the most dynamic IT companies in Asia. The company distributes a wide range of IT products across various areas including cyber security, cloud, data protection, end points, infrastructures, system monitoring, storage, networking, business productivity and communication products.

Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, different vertical industries, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About Portnox
Portnox provides simple-to-deploy, operate and maintain network access control, security and visibility solutions. Portnox software can be deployed on-premises, as a cloud-delivered service, or in hybrid mode. It is agentless and vendor-agnostic, allowing organizations to maximize their existing network and cybersecurity investments. Hundreds of enterprises around the world rely on Portnox for network visibility, cybersecurity policy enforcement and regulatory compliance. The company has been recognized for its innovations by Info Security Products Guide, Cyber Security Excellence Awards, IoT Innovator Awards, Computing Security Awards, Best of Interop ITX and Cyber Defense Magazine. Portnox has offices in the U.S., Europe and Asia. For information visit http://www.portnox.com, and follow us on Twitter and LinkedIn.。

Achieving Sarbanes-Oxley (SOX) Compliance Using Cybersecurity Controls

The Sarbanes-Oxley Act (SOX) is primarily associated with business transparency and the use of accounting and financial controls to protect investors from fraudulent financial reporting. However, it is always important to remember the ever-increasing pivotal role cybersecurity plays in SOX as digitization continues to accelerate and cybersecurity threats, financial reporting, and auditors intersect.

After all, financial data is sensitive and the financial industry has seen increasing attacks from threat actors in 2020, increasing by 238% in 2020 alone.

Additionally, the 2021 Gartner Hot Spots report names cyber vulnerabilities as a primary area of risk that auditors need to address, stating that the threat has been further amplified by “large-scale remote work.”

With regulators taking these new and emerging threats to investors into consideration, companies and auditors need to be aware of evolving requirements to keep up with SOX compliance and cybersecurity practices to protect themselves from risks like these.

Even companies that do not operate in the US or engage with US clients should take note as SOX is becoming increasingly global, with the UK Financial Reporting Council (FRC) working on a UK equivalent.

Read on to find out what you need to do to achieve Sarbanes-Oxley compliance using cybersecurity controls.

What is SOX Compliance?

The Sarbanes-Oxley Act was introduced in the US in 2002. Congressmen Paul Sarbanes and Michael Oxley merged compliance law to improve corporate governance and accountability. This was done as a response to some of the big financial scandals that took place in previous years.

The details of SOX compliance are complex. SOX compliance refers to annual audits that take place at publicly traded companies, within which they are required by law to show evidence of accurate and secure financial reporting.

These companies are required to comply with SOX both financially and IT. IT departments were affected by SOX as the Act changed the way corporate electronic records were stored and handled.

SOX’s internal security controls require data security practices and processes and complete visibility into interactions with financial records over time. Failure to comply with SOX is a serious matter, often resulting in large fines or potentially imprisonment for those responsible for the organization.

Who must comply with SOX compliance?

All publicly traded companies in the US must comply with the SOX, as well as any wholly-owned subsidiaries and foreign companies that are publicly traded and do business with the US.

Any accounting firms that are auditing firms bound by SOX compliance are also, by proxy, required to comply. Other companies, including private and non-profits, are generally not required to comply with SOX, although adhering to it is good corporate governance practice.

There are reasons other than good business sense to comply with SOX even if your company is not listed on a stock exchange. SOX has some articles that state that if any company knowingly destroys or falsified financial data, it can be punished according to the law.

Companies planning to go public, perhaps through an IPO (Initial Public Offering), should prepare to commit to SOX.

What are the benefits of SOX compliance?

SOX provides the framework companies need to follow to better manage their financial records, which in turn improves many other aspects of the company.

Companies that comply with SOX report that their finances are more predictable, which makes shareholders happy. Companies also report that they have easier access to capital markets due to improved financial reporting.

By implementing SOX, companies are safer from cyberattacks and the costly consequences of a data breach. Data breaches are difficult to manage and remediate, and companies may never recover from the damage to their brands.

SOX compliance builds a cohesive internal team and improves communication between teams involved in audits. The benefits of a company-wide program like SOX can have other tangible effects on the company – such as better communication and cross-functional cooperation.

In short, the benefits of SOX compliance are:

  • A reinforced control environment
  • Improved documentation
  • Greater involvement of the Audit Committee
  • Convergence opportunities
  • Standardized processes
  • Reduced complexity
  • Minimization of human error

What is the role of cyber security in SOX?

Companies need to remember that the scope of SOX only includes financial controls and therefore testing is limited to financial applications, servers, operating systems and databases within the scope of production.

There are many other servers and devices not reviewed for SOX compliance that could be compromised and in turn affect financial reporting. Thus, it is critical to take a holistic approach to security and internal audit that includes prevention, detection, and corrective controls to address cybersecurity risks.

Initially, internal auditors should incorporate cyber risks into their annual audit risk assessments and should interview key cybersecurity officials during the process. Now that boards are asking more questions about cyber risk and mitigation efforts, there is value in scheduling these meetings even more frequently.

Once cyber risks are identified and controls are designed, it is important to base your organization’s cyber and SOX controls with a cybersecurity framework such as those provided in the NIST Cybersecurity Framework to test and monitor the effectiveness of mitigation efforts.

The IT controls that companies review in SOX can be used across other applications and IT environments to strengthen their cybersecurity posture, including:

  • Using least privilege for access control.
  • Change network, application, firewall, database, and operating system administrator passwords regularly.
  • Password controls.
  • Restrict service accounts to only those with necessary privileges.
  • Segregation of Duties in Change Management and Access Modification.
  • App access review and certification.
  • Change management procedures.
  • Backup Procedures.

For direct evidence of SOX, companies must complete a SOX cybersecurity memorandum annually and consider additional controls. A cybersecurity memo should be completed by both internal and external IT auditors to assess how prepared the company is for a cyberattack.

These discussions often lead to how a company’s IT security and internal audit groups can benefit from each other. Based on the cyber discussions, obvious design gaps should be addressed, including issues such as limited cyber resources, lack of cyber risk assessment, lack of cyber maturity framework, poor cyber policies and procedures, inadequate cyber training, and understanding of the current state of the world. cyber program.

Disaster recovery is also starting to appear as a key SOX control, despite being historically seen as a corrective control and later outside the scope of SOX. The addition of this control includes additional focus on whether companies can recover their in-scope financial applications in the event of a cyberattack.

How to conduct a cybersecurity controls audit on SOX?

Auditing a company’s internal security controls is often the largest, most complex, and time-consuming part of a SOX compliance audit. This is because internal controls include all of the company’s IT assets such as workstations, hardware, software, and all other electronic devices that can access financial data.

SOX IT audits are focused on the following key areas:

Risk assessment and materiality analysis

Your organization needs to do a rigorous risk assessment that takes into account cybersecurity risks that fall under SOX. This approach will require cybersecurity expertise on audit teams and should also include executive and board-level information to help determine your organization’s definition of “material” cybersecurity risk.

To ensure you are covering a large number of bases, cybersecurity best practices recommend that you perform cybersecurity risk management using common frameworks like NIST and COSO to help you through the process.

When carrying out risk assessments, auditors should always examine how comprehensive and well-documented they are, as risk assessments are one of the key spheres that regulators and supervisory bodies will examine.

Fraud risk assessment

Make sure your organization has performed a thorough risk assessment for potential fraud activity to help with early detection and fraud prevention. The internal controls you are implementing should help prevent fraud and mitigate material impacts if they occur.

Implementing cybersecurity controls

After performing a risk assessment in which you have identified the cybersecurity risks, policies, and control solutions needed to comply with SOX, your company must implement these controls following industry standards.

Again, cybersecurity best practices recommend using a trusted framework such as the NIST Cybersecurity Framework (NIST CSF) as a foundation for designing Cyber SOX controls when starting to build a control environment.

Part of the implementation process will be training control owners on the purposes and reasons for controls and how they should communicate if a control fails or requires adjustment due to changes in the environment.

Monitoring and testing controls

Organizations should monitor and test the security controls they have implemented, performing periodic self-assessments, attestations, and other self-certifications. Audit teams can be a valuable resource in assessing the effectiveness of management programs and even provide practical, actionable areas to improve resilience if trained with this in mind.

It is important that you are regularly testing controls and continually monitoring the security of your own infrastructure and that of your vendors to prevent and prevent data breaches, data leaks and cyber threats. Having an understanding of log management is important in this process.

Reports

It is important that staff and auditors are familiar with the SOX disclosure requirements, knowing the correct forms of communication and the steps needed to make timely and appropriate disclosure in the event of something like a data breach.

Defining communication guidelines and who needs to be informed is a key part of incident response preparation.

What are the penalties for non-compliance with the SOX?

Being deemed non-SOX compliant can include penalties such as:

  • Fines.
  • Removal of public stock exchanges.
  • Invalidation of civil liability insurance policies for directors and executives (D&O).

There are a number of sections that outline the penalties for being found to be non-compliant with SOX, such as:

  • Section 906, where filing and certifying a misleading or fraudulent financial report can incur fines of up to $5 million and result in a criminal penalty of 20 years in prison.
  • Section 802, where altering, falsifying, destroying or concealing financial records, documents or tangible objects to obstruct, impede or influence legal investigations can incur penalties of up to 20 years in prison. It also carries a penalty of up to 10 years in prison for accountants, auditors or others who deliberately violate the requirements of maintaining all audit or review papers for a period of 5 years.
  • Section 806, where whistleblower complaints are protected from retaliation, further authorizes the US Department of Justice to criminally prosecute employers who retaliate against the respective individuals.

For IT departments and executives, SOX compliance is an important ongoing concern. However, SOX compliance is more than just passing an audit. This aspect involves defining data governance processes and procedures and a series of tangible benefits for your business.

According to a 2019 survey:

  • 57% of organizations benefit from improved internal controls over the financial reporting framework.
  • 51% have an improved understanding of control design and the operational effectiveness of the control.
  • 47% saw continuous improvement of business processes.

What are the key SOX compliance challenges for cybersecurity?

One of the biggest challenges is privileged users, who are often important and trusted company employees – the kind that don’t like to be questioned for potential fraudulent activity. To lessen the likelihood of this kind of necessary and uncomfortable questioning, IT departments often manage privileges by restricting and segregating them. Unfortunately, by restricting admin permissions, organizations are indirectly limiting productivity.

Monitoring privileged user database access is difficult, as the monitored users themselves often have the credentials needed to “beat the system” by deleting fraudulent logs they do not want to be seen. Again, however, restricting these credentials undermines efficiency, as administrators often use the database’s logging capabilities as a debug mechanism.

Another difficulty involves the need to audit access failures, whether they are invalid login attempts or unsuccessful attempts to retrieve privileged files. Either way, these types of activities are possible warning signs of fraudulent activity and must be tracked to satisfy SOX’s audit controls.

Additional challenges include monitoring schema modifications to ensure the veracity of the data structures being audited and monitoring privilege changes to maintain visibility into the user directory. It is also important to audit access to sensitive data tables and systems, such as SQL server events.

Other obstacles preventing SOX compliance for IT systems include insufficient database logs, ineffective data reporting, and poor event alerts.

The need to replay events by identifying key happenings in audit trails, archiving each event for future audits, ensuring the security of audit logs, producing scheduled reports for auditors, and being constantly aware of potential warnings of fraudulent activity (such as repeated login attempts failure) makes life more difficult for IT administrators.

Privileged Access Management as a solution to SOX Compliance

Muitos, senão todos os controles gerais de TI da SOX estão associados ao gerenciamento de acesso. Por exemplo, se a configuração de um aplicativo fizer parte de um controle de TI, saber quem fez a configuração (até o ponto de auditoria) é essencial para manter fortes controles.

A pessoa que configura os aplicativos e sistemas é um usuário privilegiado e possui acesso administrativo ao sistema. A partir dessa posição privilegiada, ela pode adicionar, editar ou excluir contas ou alterar configurações que afetam as transações financeiras.

Por exemplo, pode haver controle sobre quem pode lançar ativos no balanço patrimonial. Se esse controle puder ser manipulado sem o conhecimento de ninguém, os dados financeiros poderão ser corrompidos, e isso pode ser não intencional ou deliberado. Esta é uma receita para fraudes graves.

Many if not all of SOX’s general IT controls are associated with access management. For example, if the configuration of an application is part of an IT control, knowing who did the configuration (up to the point of auditing) is critical to maintaining strong controls.

The person who configures the applications and systems is a privileged user and has administrative access to the system. From this privileged position, the employee can add, edit or delete accounts or change settings that affect financial transactions.

For example, there may be control over who can post assets on the balance sheet. If this control can be manipulated without anyone’s knowledge, financial data could be corrupted, and this could be unintentional or deliberate. This is a recipe for serious fraud.

Companies that do not manage access well face some problems. In addition to an increased risk of cybersecurity breaches, there is also the likelihood that the SOX auditor will deem IT controls inappropriate.

A PAM (Privileged Access Management) solution provides a secure and simplified way to authorize and monitor all privileged users for sensitive systems, including systems involved in financial reporting.

PAM grants and revokes privileges to users for systems on which they are authorized. In addition, the solution centrally and quickly manages access to the type of heterogeneous systems that handle financial transactions and reports (e.g. General Ledger, ERP, Billing, banking APIs and others.)

The PAM solution creates an unalterable audit trail for any privileged operation. This feature facilitates the SOX evidence and audit process.

Benefits of the senhasegura solution for SOX compliance

We offer a PAM solution to achieve SOX compliance in the IT department and beyond.

The senhasegura solution combines robust PAM features with unique ease of installation and use. An agentless architecture simplifies deployment and ongoing changes, while other PAM solutions require the installation of a dedicated software agent on each system where privileged access is being managed.

Ease of use and installation provide major benefits for SOX compliance. The Act has the potential to constrain agility if controls are too tight and IT needs to be able to modify systems to keep up with business changes.

The senhasegura solution reinforces the internal controls and reporting requirements necessary for SOX compliance, going far beyond simply meeting the rules to implement an “inside-out” security approach to become part of your organization’s DNA.

For more information on how the senhasegura solution can help your company achieve SOX compliance, request a demo!

About Version 2 Digital

Version 2 Digital is one of the most dynamic IT companies in Asia. The company distributes a wide range of IT products across various areas including cyber security, cloud, data protection, end points, infrastructures, system monitoring, storage, networking, business productivity and communication products.

Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, different vertical industries, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About Segura®
Segura® strive to ensure the sovereignty of companies over actions and privileged information. To this end, we work against data theft through traceability of administrator actions on networks, servers, databases and a multitude of devices. In addition, we pursue compliance with auditing requirements and the most demanding standards, including PCI DSS, Sarbanes-Oxley, ISO 27001 and HIPAA.

Is Litigation Hold a Reasonable Replacement for Backup in Microsoft 365?

We get asked this question often, and at face value, it’s easy to see how one could equate litigation hold with backup – both have something to do with ‘preserving’ data. However, the reality is that backup and litigation hold differ on many points, and any company that fails to understand the differences between them (and the utility of each) will eventually learn the repercussions the hard way. Let’s explore the key differences between litigation hold and backup.

What Is Litigation Hold?

The term ‘litigation hold’ comes from US case law (2003, Zubulake v. UBS Warburg) where the judge ruled: ‘once a party reasonably anticipates litigation, it must suspend its routine document retention/destruction policy and put in place a ‘litigation hold’ to ensure the preservation of relevant documents.’

In 2010, Microsoft introduced a litigation hold (sometimes referred to as legal hold) retention feature for Microsoft Exchange to support eDiscovery. The feature was intended primarily as a way of preserving data should there be a legal need to preserve it for access and viewing during a litigation. Think of it as being for documentation purposes, not as a way to restore data back in place to operating platforms like Microsoft 365.

Microsoft later added the ability to create what they call in-place holds, which are holds based on a query (such as “find all messages containing the phrase ‘Project Starburst’). The back-end implementation of litigation and in-place holds are slightly different; you can see more details in Microsoft’s documentation.

Let me say it again, slightly differently: Litigation hold wasn’t designed with the intention of serving as a backup service. Yet, some still try to rely on it as a backup solution, particularly to make ends meet when not having a designated data security plan (including a third-party backup solution), with the reasoning that “some sort of data preservation is better than none, right?”

However, there are many drawbacks and substantial risks associated with these types of setups that lead to a risky, false sense of data security. Some of the shortcomings and risks of relying on litigation hold as a backup are:

  • Data storage quotas capped at only 110 GB
  • Some eDiscovery features require additional-cost licenses; if you don’t buy the licenses, you can’t use the features
  • User mailbox data is only kept while an Exchange Online license is assigned to the user. When a user leaves or becomes inactive, removing the license will eventually remove the data.
  • Recovering data needs an administrator and is a time-consuming process
  • The held data is not physically separate from the original copy

The bottom line is that you can’t depend on litigation hold or in-place holds as mechanisms for general-purpose recovery from mistakes or disasters. That’s not what they’re meant for, and you run the risk of losing data if you try to use them for that purpose.

What Is Backup?

Backup, by definition, provides one or more additional copies of your data, stored in a location physically separate from that of your primary dataset. Physical separation is a fundamental facet of backup, since storing your backup data in the same location as the primary data represents a single point of failure.  Effectively, there is no data redundancy in these types of setups.

With traditional on-premises backup, the physical separation rule meant having an off-premises backup stored in another building – so that in the event of a disaster, e.g. a fire in one building, would not destroy all your data. For cloud backup, it’s fair to ask ‘what cloud does my backup data go to?’ The answer is usually either ‘Microsoft Azure’ or ‘Amazon Web Services.’ Ideally, you want that data going to a cloud not operated by your SaaS application vendor (so, it wouldn’t be fair to put your Microsoft 365 data into Azure); otherwise, you’re violating the physical-separation rule.

Any service that is not providing this separation of copies is not—and should not be—considered a true backup.

At Keepit, we talk a lot about the ‘3 Ms’ that can cause data loss: mistakes made by people; mishaps at the SaaS application vendor; and malicious actions from inside or outside the organization.

Following data protection best practices, a properly executed backup scheme provides against all three of the Ms if anything should happen to the primary (original) dataset: malicious action in the form of a ransomware attack or a disgruntled employee; mistakes where someone with legitimate access accidentally deletes important data (or needs to back out changes they didn’t want to keep); and mishaps, where the service provider has an outage or data loss. Litigation holds can’t protect you against all 3 of the Ms: there’s no physical separation, limited ability to do large-scale restores, and no real concept of version control.

What to Look for In a Cloud SaaS Backup Solution

Besides the must-have features of data redundancy and availability, a worthy backup solution will offer a multitude of convenience and productivity-boosting tools and services, further distancing it from litigation hold. The first thing to look for is a solution that’s purpose-built for the cloud, not a refurbished or reskinned on-premises solution. Rather, a good, dedicated third-party backup solution.

Here are some of the key benefits to look for in a dedicated third-party backup solution:

  • Simple, quick restoration of the data you need, when and where you need it, in the format you need it
  • Direct restore from live storage, with no waiting for offline or near-line storage
  • An intuitive interface for quickly and easily finding and previewing specific files or messages before storing them
  • Secure, immutable storage in an independent cloud
  • Flexible geographic storage options to cover your data sovereignty requirements
  • A predictable and transparent cost model, with no hidden surprise charges for data ingress, egress, or storage

For more insight into data protection in the cloud era, get an in-depth look via the e-guide on Leading SaaS Data Security. Or, if you’d like to learn more about Keepit backup and recovery services for Microsoft 365, Salesforce, Google Workspace, and others, visit this page.

About Version 2 Digital

Version 2 Digital is one of the most dynamic IT companies in Asia. The company distributes a wide range of IT products across various areas including cyber security, cloud, data protection, end points, infrastructures, system monitoring, storage, networking, business productivity and communication products.

Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, different vertical industries, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About Keepit
At Keepit, we believe in a digital future where all software is delivered as a service. Keepit’s mission is to protect data in the cloud Keepit is a software company specializing in Cloud-to-Cloud data backup and recovery. Deriving from +20 year experience in building best-in-class data protection and hosting services, Keepit is pioneering the way to secure and protect cloud data at scale.

Surely nobody would write a web service in C++

A while back, one of my colleagues was hanging out in an online developer forum and some people there were starting up a good old-fashioned language war (the type of exchange where one person familiar with language A will announce its superiority over language B, with which the other person isn’t familiar – not really a productive use of time when you think about it, but a popular pastime nonetheless).

During this debate, one developer confidently proclaims that ‘surely nobody would ever write a web service in C++,’ to which my colleague responds, ‘well that’s exactly what we did here at Keepit.’ This prompted some questions, and this piece is an attempt to explain why we did what we did and to explain how this choice has been working out for us, given that this code base started life about 10 years ago.

To put things into perspective, it will be necessary to start with some minimal background information about this service we set out to build.

What is Keepit?

Keepit is a backup service in the cloud. We will store backup copies of your (cloud) data so that if—or when—your primary data is compromised for one reason or another (but most likely because of ransomware or account takeover via phishing), then you will still have complete daily copies of your data going back as many years as you want.

Years of data. This should make you think.

Several years ago, Microsoft claimed having 350 million seats on their 365 platform, which is one of the cloud platforms that we protect. Let’s say we get just 10% of that market (we should get much more because we are by far the best solution out there, but let’s be conservative for the sake of argument), that means we need to store all data for 35 million people (and that’s just on one of these platforms – we protect several other platforms as well).

It doesn’t end there: being backup, we copy all your changes, and we hold your old data, and that means when you clean up your primary storage and delete old documents, we keep the copy. Many customers want a year or three of retention, but we have customers who pay for 100 years of retention.

One hundred years. That means our great grandchildren will be fixing the bugs we put in our code today. This should make you think too.

Knowing the very high-level goals of our service, let’s talk about requirements for such a thing.

Core system: storage

We knew from the get-go that we would be implementing a storage solution which would need to store a very large amount of data (as everything is moving to the cloud, let’s say a few percent of all the world’s data) for a very long period of time (say a hundred years).

Now everyone in the storage business will talk about SSDs, NVMe, and other high-performance data storage technologies. None of this is relevant for large scale, affordable storage, however. Spinning disks is the name of the game and probably will be for at least another decade.

SSDs are getting the density, sure, but they are still not close to viable from a cost perspective. This means we will be writing all data to rotating magnetic media. When you write data to magnetic media, over the years, your media will demagnetize. That means, if we store a backup on a hard drive today, we probably can’t read it back just ten years from now.

That means we need to regularly move all this data from system to system to keep the data ‘fresh.’ Talking about performance, large capacity hard drives today rotate at 7200rpm, exactly the same speed as back in 1992. Access time is dominated by the rotational latency, which means that this is really an aspect of computers that has been almost at a standstill for 30 years while everything else has become faster in every way. We knew we had to deal with this.

I should probably note here that yes, we are talking about running our software on actual physical computers – no public cloud for us. If you want to go big, don’t do what the big players say you should do, do what the big players do. If public cloud was so great, Microsoft wouldn’t have built their own to run 365 – they would have run it on AWS which was very well established long before Microsoft thought about building 365. This doesn’t mean you can’t prototype on public cloud of course.

To solve our core storage need, we designed a filesystem—basically, an object storage system optimized for storing very large-scale backup data. Clearly, we expect the implementation of this storage system to have a significant lifespan.

We may want to create a better implementation one day in the future when hardware has evolved far beyond what we can imagine today, but it is worth pointing out that the storage systems we use today are very similar in architecture to what they would look like 30 years ago, and I would assume in 30 years from today. Clearly, the core code that manages all of your data is not something you want to re-write every few weeks.

So, to implement this system, we went out looking for which new experimental languages had been invented in the six months leading up to implementation start. No wait, we didn’t.

What we need from a language

There are really two types of languages:

1: Systems programming languages – those that have practically no runtime, where you can look at the code and have a high degree of confidence in understanding exactly what that leads to on your processor, the type of language you would write an operating system kernel in. This would be languages like C, C++, and who knows – maybe Rust or something else.

2: The higher-level languages, which often have significant runtimes. The good ones of these offer benefits that you cannot get in a language without a significant runtime. This would be a language like Common Lisp, but people more commonly talk about C# and Java even though I will argue they only do so because nobody taught them Lisp.

And then you have the other languages that fit various niche use cases. This could be Python, Haskell, JavaScript, and so forth. I don’t mean to talk them down, but they are not reasonable languages for software development of the type we are talking about here; and since what we’re talking about here isn’t actually so special, you could take my argument to mean that they are just not very reasonable languages for software development outside of niche uses, and that would be a fair interpretation of my opinion.

So, to be a little more concrete, what it is that we really need from a language is:

1: It must support the efficient implementation of algorithms and data structures; meaning we must have tight control over memory when we need it, our language must support the actual hardware data types like 64-bit integers on modern processors, etc. So, this rules out Python (not compiled), Ruby (not compiled) and JavaScript (JIT but doesn’t have integers or arrays).

2: When we write code today, the tool chain in 20 years’ time must still support our code with little or no changes. Because we simply can’t rewrite our code every few years. We will get nowhere if that’s what we do. That’s why large, important software systems today are still often written in C – because they started out life in the 80s or 90s and they are still the most significant operating system kernels or database management systems that exist to this day.

Steady evolution is the recipe, not rewrite from scratch every three years. This basically rules out any language that hasn’t been standardized and widely used for at least 10 years before we start the project. Meaning, since we started in 2012, that rules out any language that came out after 2002, so Go, Rust, and many other languages would have been out of the picture. C and C++ would work though.

3: We run on Linux. If you do anything significant with computers on a network, you probably run on Linux, too. We don’t want a language that is ‘ported’ to Linux as a curiosity – like C#. We need a language that is native on Linux with a significant and mature toolchain that is certain to receive significant investment for decades to come. Again, that’s really C and C++.

4: You need to design for failure. Everything from writing to a disk, to allocating the smallest piece of memory, can and will eventually fail. Relying on the developer to check error codes or return values at every single call to a nontrivial function (and too many trivial functions too) is rough. Yes, it can be done and there are impressive examples of this.

I am humbled by software such as the Postgres database or the Linux kernel which are very reliable pieces of software written in C which require such tedious checking. C++, in my experience, with RAII and exceptions, offers a much safer alternative. It is not free, of course – it avoids one set of problems and introduces another. In my experience however, it is less difficult to write reliable software using RAII and exceptions than to rely on developers not missing a single potential error return and correct recovery and cleanup. For this reason, I will prefer C++ over C and over both Rust and Go.

5: Obviously the language must offer sufficiently powerful functionality to make the implementation of a larger application bearable and maybe even enjoyable. In reality, however, if your language has functions, you can accomplish a lot; Fortran got functions in 1958 and since then most languages have had them.

Yes, generic programming is nice in C++. A real programmable language like Common Lisp would be preferable of course. Any other modern programming language will surely have some other feature which was added because it is potentially nice and potentially justifies the existence of the language.

But in reality, the hard part is getting your data structures right. Getting your algorithms right. Knowing what you’re trying to build and then building exactly that, nothing more and nothing less.

If we are honest, most languages would work. However, C++ is a nice compromise: it has some generic programming, the STL is incredibly useful, it offers basic OO concepts, and RAII (and structured error handling).

If we look at the criteria here, there really aren’t that many candidate languages to choose from, even if we compromise a bit here and there. Therefore, the question really isn’t ‘why’ we would write a web service in C++, the question really is ‘why wouldn’t we’ write a web service in C++. Realistically, what else would you use, given the scope of what we’re solving here?

Versatility

Performance matters. Don’t let anyone tell you otherwise. Anyone who says that ‘memory is cheap’ and uses that as an excuse should not be building your large-scale storage systems (or application servers or anything else that does interesting work on large amounts of data).

Donald Knuth said, ‘Premature optimization is the root of all evil’ and I absolutely believe that. However, ‘no optimization and elastic scaling is the root of all public cloud revenue’ is probably also true. Don’t go to extremes – don’t put yourself in a situation where you cannot, at the appropriate time, optimize your solution to be frugal with its resource use. When your solution is ‘elastically scaling’ for you in some public cloud on a credit card subscription, it is very hard to go back and fix your unit economics. Chances are you never will.

The typical computer configuration for a storage server in Keepit is 168 18 TB hard drives attached to a single-socket 32-core 3.4GHz 64-bit processor and 1TiB of RAM. It’s really important to note here that we use only one TiB of RAM for three PiB of raw disk: this is a 3000:1 ratio – it is not uncommon to see general purpose storage systems recommend a 30:1 ratio of disk to RAM (which would require us to run with 100TiB of RAM at which point memory most certainly isn’t cheap anymore). Through the magic of our storage software, this gives us about 2PiB of customer-usable storage in only 11U of rack space. This means we can provide a total of 8PiB of usable storage in a single 44U rack of systems, consuming less than 10kW of power. This matters.

If you run a business, you want to be able to make a profit. Your customers will want you to make a profit, especially if they bet on you having their data 100 years from now. If you want to grow your business with investments, your investors will think this matters. In Keepit, we have amazing unit economics – we got the largest series A round of investment for an IT company in the history of Denmark ever – and part of the reason for that was because of our unit economics. Basically, our storage technology, not least the implementation of it, enabled this.

The choice of C++ has allowed us to implement a CPU- and memory-efficient storage system reliably that uses the available hardware resources to their fullest extent. This ranges from careful layout of data structures in memory to an efficient HTTP stack that exposes the functionality and moves more than a GiB of data per second per server over a friendly RESTful HTTP API on the network. C++ enables and supports every layer of this software, and that is quite a feat.

Let me briefly digress with another note on versatility. I have this personal hobby project where I am developing a lab power supply for my basement lab (because every basement needs a lab). In order to adjust current and voltage limits, I want to use rotary encoders rather than potentiometers.

A rotary encoder is basically an axle that activates two small switches in a specific sequence and by looking at the sequence you can detect if the user is turning the axle in one direction or the other. The encoder signal gets fed to a 1MHz 8-bit processor with 1 kB of RAM and 8 kB of flash for my code.

To implement the code that detects the turning of these encoders, it makes sense to use a textbook, object-oriented approach. Create a class for an encoder. Define a couple of methods for reading the switches and for reading out the final turn data. Declare a bit of local state. Beautifully encapsulated in pure OO style. The main logic can then instantiate the two encoders and call the methods on these objects. I am implementing the software for this project in C++ as well – try to think about that for a moment: The same language that allows us to efficiently and fully utilize a 32-core 3.4GHz 64-bit processor with 1TiB of RAM and 3PiB of raw disk works ‘just as well’ on a 1-core 1MHz 8-bit processor with 1kiB of RAM and 8kiB of flash storage – and the code looks basically the same.

There are not many languages that can stretch this wide and not show the slightest sign of being close to its limit. This is truly something to behold.

The rest of the stack

The storage service exposes a simple RESTful API over HTTP using an HTTP stack we implemented from scratch in C++. Instantiating a web server in C++ is a single line of code – processing requests is as trivial as one could wish for.

I’ve heard plenty of arguments that doing HTTP or XML or other ‘web’ technology work would be simpler in Java or C# or other newer languages, but really, if you write your code well, why would this be difficult? Why would you spend more than a line of code to instantiate a web server? Why would parsing an XML document be difficult?

For XML, we implemented a validating parser using C++ metaprogramming; I have to be honest and say this was not fun all the way through and I couldn’t sit down and write another today without reading up on this significantly first. C++ metaprogramming is nothing like a proper macro system – but it can absolutely solve a lot of problems, including giving us an RNC-like schema syntax for declaring a validating XML parser and generating efficient code for exactly that parser.

This also means when we parse an XML document and we declare that one of the elements is an integer, then either it parses an integer successfully or it throws. If we declare a string, we get the string properly decoded so that we always work on the native data – we cannot ever forget to validate a value and we cannot ever forget to escape or un-escape data. By creating a proper XML parser using the language well, we have not only made our life simpler, we have also made it safer.

The entire software ecosystem at Keepit may revolve around our storage systems, but we have several other supporting systems that use our shared components for the HTTP and XML stack.

One other notable C++ system is our search engine. Like so many other companies, we found ourselves needing a search engine to assist us with providing an amazing end user experience when browsing their datasets. And like so many others we fired up a cluster of Elasticsearch servers and went to work.

Very quickly we got hit by this basic fact that Elastic is great at queries and not very good at updates – and we have many more updates than we have queries. We simply couldn’t get this to scale like we’re used to. What to do?

While struggling with Elastic, we started the ‘Plan-B’ project to create a simple search engine from scratch – this engine has been our only search engine for years now and to this day, the process is still called ‘bsearch.’

Our search engine offers a google-like matching so that you can find your documents even if you misspell them, and it is a piece of technology that we are quite actively developing both to improve matching capabilities across languages and to allow for more efficient processing of much larger datasets, which will open up for other uses in the future.

Of our backend code base, about 81% of our code is C++. Another 16% is Common Lisp. The remaining 3% is Java.

We use Common Lisp in two major areas: For ‘general purpose’ business functions such as messaging, resource accounting, billing, statistical data processing, etc. And we use it for backup dataset processing. These are two very different uses.

The first is a more classical application of the language where performance is maybe less of a concern but where the unparalleled power of the language allows for beautiful implementations of otherwise tedious programs.

The second use is a less traditional use case where enormous datasets are processed and where the majority of the memory is actually allocated and managed outside of the garbage collector – it is truly a high-performance Lisp system where we benefit from the power of the language to do interesting and efficient extractions of certain key data from the hundreds of petabytes of customer data that pass through our systems.

Many people don’t know Common Lisp and may propose that ‘Surely nobody would write a web service in Common Lisp.’ Well, as with all other languages you need to understand the language to offer useful criticism; and the really groundbreaking feature of Common Lisp is its macro system. It is what makes Common Lisp by far the most powerful language in existence by a large margin.

This is nothing like C pre-processor macros; the Common Lisp macro system allows you to use the full power of the language to generate code for the compiler. Effectively, this means the language is fully programmable. This is not something that is simple to understand since there is no meaningful way to do this using C-like language syntax, which is also why the Lisp dialects have a syntax that is fundamentally different from other languages.

In other words, if you do not understand the Lisp syntax, you are not equipped to comprehend what the macro system allows. This is not simple to wrap your head around, but, for example, I can mention that Common Lisp was the first general purpose programming language to get Object Orientation added to it, and this was done not with a change to the language and the compiler, but with a library that contained some macros. Imagine that.

Fortran allows you to implicitly declare the type of variables by using certain letters in the first character of the variable name – just for fun, I implemented that with a macro for Common Lisp. If I wanted to do that with C or C++ or any other language, I would need to extend the compiler.

The idea of using the first character in the name of the variable to implicitly declare its type is of course ridiculous, but there are many little syntactical shortcuts or constructs that can help you in daily life that you may wish was present in your language of choice which you can only hope the language steering committee may one day add to the standard.

With Common Lisp, this is everyday stuff – no need to wait. If you want a new type of control structure or declaration mechanism, just go ahead and build it. The power of this cannot be overstated. C++ metaprogramming (and go generics and everything else) pales in comparison, useful as it is.

Lessons learned

First of all, it really sucks to have multiple languages; you can’t expect everyone to be an expert in all, so by having more than one language, you decimate the effective size of your team. However, we picked Common Lisp to replace a sprawling forest of little scripts done in more languages than I could shake a stick at—meaning we are fortunate to have only two languages on our backend.

C++ and Common Lisp are so different that they complement each other well. Yes, we could have done everything in C++, but there are problems we solve in Common Lisp which would have been much less enjoyable to solve in C++. Now on the downside, we have two HTTP stacks, two XML stacks, two database libraries, two connection pools, and so on and so forth. There is no simple perfect solution here; the compromise we have arrived at is indeed working out very well for us.

We’ve been told many times that recruiting for C++ is hard because recruiting for ‘web technologies’ is so much simpler. Well guess what, finding good JavaScript developers is just as hard as finding good C++ developers in my experience. With Common Lisp it’s different again: it’s harder to find people, but the percentage of the candidates that are actually qualified is higher, so all in all, it’s actually fine. Recruitment is difficult across languages, period.

The best you can do is go to a conference, talk about your tech, and hope that some developers show up at your booth to talk about employment.

Old grumpy man’s advice for youngsters considering a career in software engineering

First of all, seriously consider a computer science education. There exist amazingly qualified people who do not have this and some of them work for us, but in my experience most really good developers have this. It certainly helps to get a foundation of mathematics, logic, and basic computer science. Knowing why things work will make learning new things infinitely simpler.

Learn multiple, properly different programming languages and write actual code in them. You need to experience (by failing) how functions are useful as abstractions and how terrible it is to work with ill-designed abstractions. You need to fail and spend serious time failing.

Make sure one of those languages is a compiled language with little or no runtime: C, C++, Rust, or even Fortran for that matter (not sure Fortran has much long-term perspective left in it though – it’s probably time to say goodbye). Now challenge yourself to write the most efficient implementation of some simple problem – maybe a matrix multiplication for example.

Disassemble the code and look at it. At least get some understanding of the processor instructions and why they are generated from the code you wrote. Learn how cache lines matter. Time your code and find out why your solution isn’t faster than it is. Then make it faster until you can prove to yourself that your instructions pipeline as much as they can, your cache misses are minimal, you don’t wait on register write delays and so on and so forth.

Also, make sure that one of those languages is Common Lisp. It should be a criminal offence for a university to not teach Common Lisp in their computer science curriculum. Read ‘The Structure and Interpretation of Computer Programs – SICP’ too. Even if you will never use Lisp again, knowing it will make you a better developer in any other language.

And finally, as much as I dislike JavaScript, you should learn that, too. The most beautiful backend code will too easily be ignored if you cannot beautifully present its results – and today this means doing something with JavaScript.

Aside from my previous criticisms, you can make working with JavaScript more bearable, for example, by creating your own framework rather than relying on the constantly changing megabyte sized atrocities that your common web projects rely on. However, this is probably a topic for future discussion.

About Version 2 Digital

Version 2 Digital is one of the most dynamic IT companies in Asia. The company distributes a wide range of IT products across various areas including cyber security, cloud, data protection, end points, infrastructures, system monitoring, storage, networking, business productivity and communication products.

Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, different vertical industries, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About Keepit
At Keepit, we believe in a digital future where all software is delivered as a service. Keepit’s mission is to protect data in the cloud Keepit is a software company specializing in Cloud-to-Cloud data backup and recovery. Deriving from +20 year experience in building best-in-class data protection and hosting services, Keepit is pioneering the way to secure and protect cloud data at scale.

Pandora FMS boosts its cloud solution. Safer, easier and faster IT monitoring

For their MaaS Solution (monitoring as a service), Pandora FMS   partners with OVHcloud to ensure data sovereignty, European technological independence and top quality service with datacenters in Europe, America and Asia.

Learn all about Monitoring as a Service (MaaS)

Pandora FMS, the renowned Spanish technology company, launches its SaaS solution on the market: MaaS, Monitoring as a Service, a flexible subscription model, easy to understand and that covers all the monitoring needs of companies of all sizes.

The MaaS Solution complies with easy integration with business processes, permanent security and 24/7 availability, to be accessed anytime and anywhere.

A crucial weapon to safeguard companies from the growing demand for the Cloud, and the difficulties of finding personnel to manage the ever-increasing technological complexity. 

Particularly, since it is well known that the use of heterogeneous tools and the lack of use of AI in multi-cloud environments prolongs and delays innovation, forcing IT teams to devote almost half of their time to maintenance tasks.

In collaboration with OVHcloud

Since 1999, the year of its foundation, OVHcloud has had a single objective: to offer their customers a complete range of innovative Cloud products and bare metal with a marked European accent.

Since then, the group has provided world-class infrastructures to more than 1.6 million customers worldwide, and has an ecosystem of partners who bring in their added value to accompany the digital transformation of  different companies. 

Pandora FMS is incorporated as Advance Partner from OVHcloud, combining its extensive experience in IT monitoring software with the sovereign infrastructures of OVHcloud, and includes its ready-to-use solutions in the Marketplace of the European Cloud leader.

At the moment, OVHcloud’s powerful high-end dedicated servers are helping Pandora FMS provide its customers with its MaaS service: a service with very high standards by offering 24/7 operation. 

Thanks to the smooth operation and reliability of this product, they can also comply with the SLA of the services offered hosted on these servers.

“The ease of provision, the transparency and the wide range of options that OVHcloud now has to choose, not to mention the possibility of having servers in Europe or America, is key to deploying our solutions,” tells us Sancho Lerena, CEO and founder of Pandora FMS.
“In addition, it is a pleasure to be part of the OVHcloud ecosystem, and contribute jointly to the promotion of innovative, interoperable and trusted solutions. One more way to try in face of all the range of possibilities of our MaaS Solution,” concludes.

Pandora FMS

Pandora FMS, as many of you know, is the total monitoring solution, which allows you to observe and measure all kinds of technologies regardless of where they are located: Cloud, SaaS, virtualization or on-premise: a flexible solution that unifies data visualization for full observability of the entire organization.

With more than 50 thousand facilities in 53 countries, among its customers there are companies such as:

  • Salvensen
  • Prosegur
  • Repsol
  • CLH
  • Euskaltel
  • Adif 
  • Santalucía
  • Cofares 
  • AON
  • El Pozo 
  • The EMT 

And other foreign companies such as:

  • Rakuten
  • Nazareth University in New York 
  • Ottawa’s main hospital 

Also, public administrations such as:

  • La Junta de Castilla-La Mancha 
  • Community of Madrid
  • La Diputación de Barcelona 
  • And numerous municipalities in France, Portugal and Spain

About Version 2 Digital

Version 2 Digital is one of the most dynamic IT companies in Asia. The company distributes a wide range of IT products across various areas including cyber security, cloud, data protection, end points, infrastructures, system monitoring, storage, networking, business productivity and communication products.

Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, different vertical industries, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About PandoraFMS
Pandora FMS is a flexible monitoring system, capable of monitoring devices, infrastructures, applications, services and business processes.
Of course, one of the things that Pandora FMS can control is the hard disks of your computers.

Need help? Our team is just a message away