Big Data infrastructure refers to the systems (hardware, software, network components) and processes that enable the collection, management, and analysis of massive datasets. Companies that handle large volumes of data constantly coming in from multiple sources often rely on open source Big Data frameworks (i.e. Hadoop, Spark), databases (i.e. Cassandra), and stream processing platforms (i.e. Kafka) as the foundation of their Big Data infrastructure.

In this blog, we’ll explore some of the most commonly used technologies and methods for data storage, processing, mining, and visualization in an open source Big Data stack. 

Data Storage and Processing

The primary purpose of Big Data storage is to successfully store vast amounts of data for future analysis and use. A scalable architecture that allows businesses to collect, manage, and analyze immense sets of data in real-time is essential. 

Big Data storage solutions are designed to address the speed, volume, and complexity of large datasets. Examples include data lakes, warehouses, and pipelines, all of which which can exist in the cloud, on-premises, or in an off-site physical location (which is referred to as colocation storage).

Data Lakes

Data lakes are centralized storage solutions that process and secure data in its native format without size limitations. They can enable different forms of smart analytics, such as machine learning and visualizations.

Data Warehouses

Data warehouses aggregate datasets from different sources into a single storage unit for robust analysis, data mining, AI, and more. Unlike a data lake, data warehouses have a three-tier structure for storing data.

Data Pipelines

Data pipelines gather raw data from one or more sources, potentially merge and transform it in some way, and then transport it to another location, such as lakes or warehouses.

Related Technologies

No matter where data is stored, at the heart of any Big Data stack is the processing framework. One prominent open source example is Apache Hadoop, which allows for the distributed processing of large datasets across clusters of computers. Hadoop has been around for a long time, but is still popular especially for non-cloud-based solutions. It can be seamlessly coupled with other open source data technologies like Hive or HBase for a more comprehensive implementation to meet business requirements. 

Data Mining

Data mining is defined as the process of filtering, sorting, and classifying data from large datasets to reveal patterns and relationships, which helps enterprises identify and solve complex business problems through data analysis. 

Machine learning (ML), artificial intelligence (AI), and statistical analysis are the crucial data mining elements that are necessary to scrutinize, sort, and prepare data for deeper analysis. Top ML algorithms and AI tools have enabled the easy mining of massive datasets, including customer data, transactional records, and even log files picked up from sensors, actuators, IoT devices, mobile apps, and servers.

Every data science application demands a different data mining approach. Pattern recognition and anomaly detection are two of the most well-known and both employ a combination of techniques to mine data.Let’s look at some of the fundamental data mining techniques commonly used across industry verticals.

Association Rule

The association rule refers to the if-then statements that establish correlations and relationships between two or more data items. The correlations are evaluated using support and confidence metrics, where support determines the frequency of occurrence of data items within the dataset, and confidence relates to the accuracy of if-then statements.

For example, while tracking a customer’s behavior when purchasing online items, an observation is made that the customer generally buys cookies when purchasing a coffee pack. In such a case, the association rule establishes the relation between two items (cookies and coffee packs), and forecasts future buys whenever the customer adds the coffee pack to the shopping cart.

Classification

The classification data mining technique classifies data items within a dataset into different categories. For example, vehiclescan be grouped into different categories, such as sedan, hatchback, petrol, diesel, electric, etc., based on attributes such as the vehicle’s shape, wheel type, or even number of seats. When a new vehicle arrives, it can be categorized into various classes depending on the identified vehicle attributes. The same classification strategy can be applied to categorize customers based on factors like age, address, purchase history, and social group.

Clustering

Clustering data mining techniques group data elements into clusters that share common characteristics. Data pieces get clustered into categories by simply identifying one or more attributes. Some of the well-known clustering techniques are k-means clustering, hierarchical clustering, and Gaussian mixture models.

Regression

Regression is a statistical modeling technique using previous observations to predict new data values. In other words, it is a method of determining relationships between data elements based on the predicted data values for a set of defined variables. This category’s classifier is called the “Continuous Value Classifier.”

Sequence & Path Analysis

One can also mine sequential data to determine patterns, wherein specific events or data values lead to other events in the future. This technique is applied for long-term data as sequential analysis is key to identifying trends or regular occurrences of certain events. For example, when a customer buys a grocery item, you can use a sequential pattern to suggest or add another item to the basket based on the customer’s purchase pattern.

Neural Networks

Neural networks technically refer to algorithms that mimic the human brain and try to replicate its activity to accomplish a desired goal or task. These are used for several pattern recognition applications that typically involve deep learning techniques. Neural networks are a product of advanced machine learning research.

Prediction

The prediction data mining technique is typically used for predicting the occurrence of an event, such as machinery failure or a fault in an industrial component, a fraudulent event, or company profits crossing a certain threshold. Prediction techniques can help analyze trends, establish correlations, and do pattern matching when combined with other mining methods. Using such a mining technique, data miners can analyze past instances to forecast future events.

Related Technologies

When it comes to data mining tasks, open source technologies like Spark, YARN or Oozie are great engines that use flexible and powerful Map Reduction and batching techniques.

Data Visualization

Data visualization is the graphical representation of information and data. With visual elements like charts, graphs, and maps, data visualization tools provide an accessible way to see and understand trends, outliers, and patterns in data.

As more companies increasingly depend on their Big Data to make operational and business-critical decisions, visualization has become a key tool to make sense of the trillions of rows of data generated every day.

Data visualization helps tell stories by curating data into a medium that is easier to understand. A good visualization removes the noise from data and highlights the useful information, like trends and outliers.

However, it’s not as simple as just dressing up a graph to make it look better or slapping on the “info” part of an infographic. Effective data visualization is a delicate balancing act between form and function. The plainest graph could be too boring to catch any notice, or it could make a powerful point; likewise, the most stunning visualization could utterly fail at conveying the right message or it could speak volumes. The data and the visuals need to work together, and there’s an art to combining great analysis with great storytelling.

Related Technologies

The open source software that best responds to these needs is Grafana, which encompasses all basic visualization elements. With a tool like Grafana, a business will be able to effectively monitor their Big Data implementation, and let data visualizations drive informed decisions, enhance system performance, and streamline troubleshooting.

Final Thoughts

While we’ve covered some of the fundamentals of Big Data infrastructure here, it should go without saying that there is much more to this topic than can be covered in a single blog post. It’s also worth noting that implementing and maintaining Big Data infrastructure requires a high level of technical expertise. These technologies are among the most complex, which is why companies that lack the in-house capabilities often turn to third parties for commercial support and/or Big Data platform administration. Investing in a Big Data platform can deliver big rewards, but only if it’s backed by a solid Big Data strategy and managed by individuals who have the necessary skills and experience.

Unlock the Power of Your Big Data

If you need to modernize your Big Data infrastructure or have questions about administering or supporting technologies like Hadoop, our Enterprise Architects can help.

Talk to a Big Data expert

Why Companies Need OSS Support

The Top 5 Reasons Companies Choose OpenLogic for OSS Support

For more than 20 years, OpenLogic has offered expert OSS technical support and professional services (i.e. consulting, migrations, training) to organizations around the world. Below are insights from customers sharing what made them pick OpenLogic as their OSS partner. 

1. One Vendor Who Can Support All the OSS in Your Stack

2. Consistent, Direct Support From Experienced Enterprise Architects

3. Meet Compliance Requirements With SLA-Backed Support

4. Expertise Integrating Open Source Packages Into Full Stack Deployments

5. Unbiased Guidance Regardless of Infrastructure or Environment

Keeping track of software dependencies is not an easy task and only becomes more difficult as companies scale. In this blog, we explore the types of dependencies and complications they can cause, as well as available tools and best practices organizations can adopt to improve their open source inventory management.

Understanding Software Dependencies

Software dependency management is a hot topic and an ongoing area for learning and process improvements. They are the byproduct of code collaboration and sharing, and all of us who consume and/or contribute to OSS are potential victims of the consequences if dependencies aren’t properly managed. And while not unique to open source software, the rapid proliferation of open source technologies has made tracking software dependencies more complex.

There are two main categories of software dependencies:

• Direct dependencies: This refers to frameworks, libraries, modules, and other software components that an application deliberately and “directly” references to address a solved problem.
• Transitive dependencies: This refers to the cascading list of those independent pieces of software that the direct dependencies in turn include to function properly.

Beyond that, there are some distinctions within those two main categories that are good to be aware of before defining a dependency management strategy:

• Internal vs. External: Some dependencies may be owned and controlled internally by a development team, though typically the vast majority are created and maintained externally.
• Open vs. Closed: Referenced dependencies may be open source allowing development team investigation and ownership by proxy, or they might be binary-only licensed from a vendor where changes are managed through contractual terms.
• Idle vs. Engaged: As the application source evolves, needs change, rendering some dependencies irrelevant. However, they are not always removed from the dependency chain. As a result, some dependencies are actively engaged and used, whereas others are no longer used and remain bundled but idle.

A software inspection methodology that includes inventorying dependencies and managing lifecycles is essential to system security and sustainability. An up-to-date software inventory is necessary for identifying vulnerable or end-of-life components, and identification is the first step in remediating issues and mitigating risks.

The Challenges of Dependency Management

Today there is an ever-increasing demand for both speed and innovation with regards to software development, and that is both the catalyst for, and the result of, open source software. This demand has also produced software delivery concepts like microservices and container orchestration that require vast amounts of integration points – all of which contribute to the chain of software dependencies. This has ushered in a host of software maintenance problems that require dependency management solutions.

The main challenges that arise are due to the pace of change. It is increasingly more difficult for organizations to keep up with evolving software, as well as the companies, communities, and licensing bodies that maintain and govern them. Some examples:

• Version conflicts: When multiple dependencies within the same application require different versions of a shared library.
• Compatibility issues: When updating a package can introduce breaking changes that require modifications to your application to maintain existing functionality.
• Security vulnerabilities: When a downstream dependency has a known security defect that either needs to be addressed by your application or requires an update to the dependency to remediate it.
• End-of-Life problems: When the referenced software package is no longer maintained by the vendor or community, which can result in security defects that are not remediated and leave your application vulnerable to attacks.
• License compliance: When the application uses another software component in a way that is not allowed by the software license. This can sometimes happen as the result of a license change as versions of the dependency are upgraded.
• Idle bloat: When an application has a growing number of unreferenced dependencies that increase the size, complexity, and liability without adding value.

Few developers are privy to all dependency management best practices, and most teams are not equipped with the tooling necessary to mount a proactive approach to avoid dependency problems. Gone are the days when a development team would settle on a single programming language that allowed them to use a particular package manager (e.g. python:pip, java:maven, javascript:npm, rpm:yum) to list the dependency tree, checklists to track the inventory, and unit tests to validate upgrades. Professionalizing a software development practice now requires modern systems for tackling software dependency management at scale.

Unbiased Guidance. SLA-Backed Support.

For more than two decades, OpenLogic has partnered with enterprises to help them get the most from their OSS. From migrations to technical support, we can tackle the toughest open source challenges — freeing up your team to focus on innovating for your business.

Let’s Talk

How to Track Software Dependencies and Manage Your Open Source Inventory

Unfortunately as of this writing, there is no silver bullet in this space. In fact, there is not even a best-in- class solution that has emerged. The good news is, there are software organizations and communities that recognize the problem, and are developing strong solutions to address pieces of this puzzle. Gluing them together can produce an effective system, which is the best path forward for now.

Software Dependency Management Tools

There are a few cornerstone tools that lay the foundation for a modern software dependency management system:

• A central code repository that supports revision control and release versioning (e.g. Git, Github, Gitlab, Helix Core). This is the foundation for dependency discovery, and it can also save and manage lock files that tie an application to a specific version of a dependency.
• A package manager for each programming language or platform (e.g. python:pip, java:maven, javascript:npm, rpm:yum). These tools will handle the interactions (push, pull, install, update, list, etc.) with a dependency repository.
• A Software Bill of Materials (SBOM) generator (Syft, SBOM Tool, Tern, CycloneDX). This will produce an attributed inventory of all the software components in your applications (including supplier name, component name, component author, component version, dependency relationship, governing license(s), etc.).
• A vulnerability scanner that supports scheduled detection scans and notification schemes (e.g. Trivy, Grype). This tool will schedule automatic scans that identify security issues and provide detailed reports (i.e. risk prioritization, remediation guidance) that help assess the impact to all direct and transient dependencies referenced by your application.

6 Dependency Management Best Practices

The tools above should be augmented by some best practices that can be implemented and enforced through internal policies, processes, and procedures. These six best practices are a good place to start:

1. Create a central artifact repository to capture the software inventory with key attributes, notes, and links to additional details in related systems (i.e roadmapping, issue tracking systems, risk management, contracts).
2. Define a clear dependency policy that lists acceptable and unacceptable sources and specific approved lists of software components, along with guidelines for gaining approval for components that fill new needs.
3. Establish update and upgrade policies that describe the tooling used to scan for dependency vulnerabilities and lifecycle attributes with guidance on how to prioritize, schedule, and apply/defer the scanner’s findings.
4. Develop a training curriculum to educate developers and others in the organization on the need for ongoing diligence around dependency management and the topics, tools, and techniques required to deliver and maintain a healthy application.
5. Adopt a versioning scheme (i.e. semantic versioning) that allows the organization to track the alignment of dependencies to a particular version of an internal application.
6. Require formal code reviews and testing that includes a dependency review geared toward heading off the common challenges identified above (e.g. version conflicts – idle bloat).

Final Thoughts

Software has become progressively more complex and the need for speed has driven more code-sharing and reuse. Developers have to rely on available packages to handle solved problems, so they can focus on new challenges that advance their particular mission. And unfortunately, sometimes tracking all the dependencies in those packages gets lost in the DevOps shuffle. Hopefully, this blog offers some actionable steps to make your approach to dependency and open source inventory management a little more sophisticated.

Apache Spark vs. Hadoop isn’t the 1:1 comparison that many seem to think it is. While they are both involved in processing and analyzing Big Data, Spark and Hadoop are actually used for different purposes. Depending on your Big Data strategy, it might make sense to use one over the other, or use them together.

In this blog, our expert breaks down the primary differences between Spark vs. Hadoop, considering factors like speed and scalability, and the ideal use cases for each.

What Is Apache Spark?

Apache Spark was developed in 2009 and then open sourced in 2010. It is now covered under the Apache License 2.0. Its foundational concept is a read-only set of data distributed over a cluster of machines, which is called a resilient distributed dataset (RDD).

RDDs were developed due to limitations in MapReduce computing, which read data from disk by reducing the results into a map. RDDs work faster on a working set of data which is stored in memory which is ideal for real-time processing and analytics. When Spark processes data, the least-recent data is evicted from RAM to keep the memory footprint manageable since disk access can be expensive.

What Is Apache Hadoop?

Hadoop is a data-processing technology that uses a network of computers to solve large data computation via the MapReduce programming model.

Compared to Spark, Hadoop is a slightly older technology. Hadoop is also fault tolerant. It knows hardware failures can and will happen and adjusts accordingly. Hadoop splits the data across the cluster and each node in the cluster processes the data in parallel very similar to divide-and-conquer problem solving.

For managing and provisioning Hadoop clusters, the top two orchestration tools are Apache Ambari and Cloudera Manager. Most comparisons of Ambari vs. Cloudera Manager come down to the pros and cons of using open source or proprietary software.

Apache Spark vs. Hadoop at a Glance

The main difference between Apache Spark vs. Hadoop is that Spark is a real-time data analyzer, whereas Hadoop is a processing engine for very large data sets that do not fit in memory.

Hadoop can handle batching of sizable data proficiently, whereas Spark processes data in real-time such as streaming feeds from Facebook and Twitter/X. Spark has an interactive mode allowing the user more control during job runs. Spark is the faster option for ingesting real-time data, including unstructured data streams.

Hadoop is optimal for running analytics using SQL because of Hive, a data warehouse system that is built on top of Hadoop. Hive integrates with Hadoop by providing an SQL-like interface to query structured and unstructured data across a Hadoop cluster by abstracting away the complexity that would otherwise be required to write a Hadoop job to query the same dataset. Spark also has a similar interface, Spark SQL, which is part of the distribution and does not have to be added later.

Get SLA-Backed Support for Hadoop or Spark

Managing a Big Data implementation can be challenging if you don’t have the right internal resources. Our Big Data experts can provide 24/7 technical support and professional services (upgrades, migrations, and more) so you can focus on leveraging the insights from your data.

Talk to a big data Expert

Spark vs. Hadoop: Key Differences

In this section, let’s compare the two technologies in a little more depth.

Ecosystem

The core computation engines of Hadoop and Spark differ in the way they process data. Hadoop uses a MapReduce paradigm that has a map phase to filter and sort data and a reduce phase for aggregating and summarizing data. MapReduce is disk-based, whereas Spark uses in-memory processing of Resilient Distributed Datasets (RDDs), which is great for iterative algorithms such as machine learning and graph processing.

Hadoop comes with its own distributed storage system, the Hadoop Distributed File System (HDFS), which is designed for storing large files across a cluster of machines. Spark can use Hadoop’s HDFS as its primary storage system, but it also supports other storage systems like S3, Azure Blob Storage, Google Cloud Storage, Cassandra, and HBase.

Hadoop and Spark include various data processing APIs for different use cases. Spark Core provides functionality for Spark jobs like task scheduling, fault tolerance, and memory management. Spark SQL allows SQL-like queries on large datasets and integrates well with structured data. It supports querying both structured and semi-structured data. The Spark Streaming component provides real-time stream processing by dividing data streams into small batches. MLlib and GraphX are libraries for machine learning algorithms and graph processing, respectively, that run on Spark.

Hadoop includes MapReduce, which is the core API for data processing in Hadoop.  The following tools can be added to Hadoop for data processing:

• Apache Hive is a data warehouse system built on top of Hadoop for querying and managing large datasets using a SQL-like language.

• Apache HBase is a distributed NoSQL database that runs on top of HDFS and is used for real-time access to large datasets.

• Apache Pig is a platform for analyzing large datasets that uses a scripting language (Pig Latin) to express data transformations.

For cluster management, YARN (Yet Another Resource Manager) is the most common approach to run Spark applications to run transparently in tandem with Hadoop jobs in the same cluster which provides resource isolation, scalability, and centralized management.

Spark does have a few more cluster management configurations than Hadoop.  Apache Mesos is a distributed systems kernel that can run Spark, and Spark also has native support for Kubernetes, which can be used for containerized deployment and scaling capabilities in Spark clusters.

For fault tolerance, Hadoop has data block replication that ensures data accessibility if a node fails, and Spark uses RDDs to reconstruct data in the event of failure.

Real-time processing and machine learning are both included with Spark. Spark Streaming natively supports real-time data processing with low latency, but Hadoop requires tools like Apache Storm or Apache Flink to accomplish this task. MLLib is Spark’s machine learning library, and Apache Mahout can be used with Hadoop for machine learning.

Features

Hadoop has its own distributed file system, cluster manager, and data processing. In addition, it provides resource allocation and job scheduling as well as fault tolerance, flexibility, and ease of use.

Spark includes libraries for performing sophisticated analytics related to machine learning, AI, and a graphing engine. The scheduling implementation between Hadoop and Spark also differs. Spark provides a graphical view of where a job is currently running, has a more intuitive job scheduler, and includes a history server, which is a web interface to view job runs.

Performance and Cost Comparison

Hadoop accesses the disk frequently when processing data with MapReduce, which can yield a slower job run. In fact, Spark has been benchmarked to be up to 100 times faster than Hadoop for certain workloads.

However, because Spark does not access to disk as much, it relies on data being stored in memory. Consequently, this makes Spark more expensive due to memory requirements. Another factor that makes Hadoop more cost-effective is its scalability; Hadoop mixes nodes of varying specifications (e.g. CPU, RAM, and disk) to process a data set. Cheaper commodity hardware can be used with Hadoop.

Other Considerations

Hadoop requires additional tools for Machine Learning and streaming which come included in Spark. Hadoop can also be very complex to use with its low-level APIs, while Spark abstracts away these details using high-level operators. Spark is generally considered to be more developer-friendly and easy to use.

Spark Use Cases

Spark is great for processing real-time, unstructured data from various sources such as IoT, sensors, or financial systems and using that for analytics. The analytics can be used to target groups for campaigns or machine learning. Spark has support for multiple languages like Java, Python, Scala, and R, which is helpful if a team already has experience in these languages.

Hadoop Use Cases

Hadoop is great for parallel processing of diverse sets of large amounts of data. There is no limit to the type and amount of data that can be stored in a Hadoop cluster. Additional data nodes can be added to address this requirement. It also integrates well with analytic tools like Apache Mahout, R, Python, MongoDB, HBase, and Pentaho.

It’s also worth noting that Hadoop is the foundation of Cloudera’s data platform, but organizations that want to go 100% open source with their Big Data management and have a little more control over where they host their data should consider the Hadoop Service Bundle as an alternative.

Using Hadoop and Spark Together

Using Hadoop and Spark together is a great way to build a powerful, flexible big data architecture. Typical use cases are large-scale ETL pipelines, data lakes and analytics, and machine learning. Hadoop’s scalable storage via HDFS can be used for storing large datasets and Spark can perform distributed data processing and analytics. Hadoop jobs can be used for large and long-running batch processes, and Spark can read data from HDFS and perform complex transformations, machine learning, or interactive SQL queries. Spark jobs can run on top of a Hadoop cluster using Hadoop YARN as the resource manager. This leverages both Hadoop’s storage and Spark’s faster processing, combining the strengths of both technologies.

Final Thoughts

Organizations today have more data at their disposal than ever before, and both Hadoop and Spark have a solid future in the realm of open source Big Data infrastructure. Spark has a vibrant and active community including 2,000 developers from thousands of companies which include 80% of the Fortune 500.

For those thinking that Spark will replace Hadoop, it won’t. In fact, Hadoop adoption is increasing, especially in banking, entertainment, communication, healthcare, education, and government. It’s clear that there’s enough room for both to thrive, and plenty of use cases to go around for both of these open source technologies.

Editor’s Note: This blog was originally published in 2021 and was updated and expanded in 2025. 

It’s a new year, which is a good time to reflect on what changed in the never-boring OSS world over the past 12 months — and what 2025 might bring. Read on to see what I expect we’ll be hearing and reading about this year in terms of open source trends.

Demand for More Data Sovereignty

More and more organizations are streaming and processing large data sets in realtime, for reasons ranging from observability into manufacturing processes and sentiment analysis of social media, to routing and processing financial transactions and training Large Language Models for AI applications.

Big Data technologies are complex, often requiring both specialized IT operations teams as well as infrastructure architects. As a result, many companies have turned to managed solutions in order to offload this work so their own teams can focus on the data and data analysis itself. However, many of these managed solutions have started adding non-optional features, requiring public cloud deployment, and dramatically increasing their pricing structure, often without transparency to their customers. Additionally, customers are running into compliance issues, as new regulatory requirements mandating how and where data is processed and stored are sometimes incompatible with these platforms.

Since many of these solutions are based on existing OSS technologies such as Hadoop, Kafka, and others, we expect to see companies rethinking their Big Data strategy, looking for ways to achieve data sovereignty by bringing their Big Data solutions in-house with open source software, and partnering with commercial support vendors as needed to aid in architecture and management.

Related >> Is It Time to Open Source Your Big Data Management? 

The Search for the Next CentOS Continues

On June 30, 2024, we saw a milestone in the Enterprise Linux ecosystem as CentOS 7 reached end of life. While a number of commercial offerings emerged to allow CentOS users to postpone their migrations, these are short-term solutions, and eventually companies will need to migrate to new distributions.

As CentOS was itself a 1-to-1 replacement for Red Hat Enterprise Linux (RHEL), this of course remains an option. However, this ignores one of the main reasons for using CentOS: the fact that you could use it without support contracts, or contract with third parties for support, often at steep discounts over Red Hat.

Several CentOS alternatives have emerged in the past few years, including AlmaLinux and Rocky Linux, providing essentially the same 1:1 OSS counterpart to RHEL that CentOS provided. Like CentOS, these distros are community-supported, and both are relatively new, with an unproven track record of support that makes some enterprise organizations nervous.

Additionally, many businesses have become increasingly security-minded in the last few years, due to a variety of CVE announcements against OSS software as well as supply chain attacks. A freely available Linux distribution is often not enough for these companies; they also need a secure baseline image to start from in order to streamline the security measures they need to take to protect their software. While commercial solutions such as RHEL, Oracle Linux, and SUSE Linux provide these, they come at substantial cost.

All of which is to say, there is still no clear victor in the so-called “Linux Wars” but as more companies migrate off CentOS in 2025, we’ll probably have a better sense of whether security or cost-effectiveness is the bigger driver based on where they end up.

Related >>How to Find the Best Linux Distro For Your Organization

Open Source AI Enters the Next Phase

AI has become the technology du jour, replacing previously trending topics such as the metaverse and cryptography. Technically speaking, most of the technology around AI today is around Large Language Models (LLMs) and Generative AI, which use statistical models in order to determine what to do next, whether that’s completing a conversational prompt, splicing together images, or other use cases.

Generative AI models require large amounts of training, with large amounts of data — which means that it falls under the umbrella of Big Data when it comes to open source. The need to keep these processes and technologies secure and performant is paramount — and just like with Big Data, the amount of expertise is spread thin.

AI is a hugely competitive market and that’s not going to change in 2025. There are a variety of toolchains already available for training LLMs and other models within Big Data pipelines, with tools such as Apache Spark, Apache Kafka, and Apache Cassandra providing key functionality used to train these models. I anticipate seeing more companies developing bespoke LLMs that directly support the products they produce, and they will use open source toolchains to do this.

Related >>Open Source and AI: Using Cassandra, Kafka, and Spark for AI Use Cases

Lessons From the XZ Utils Backdoor

In 2024, the security world was rocked by the discovery of a malicious backdoor in the xz utility, and attention was turned to staving off future supply chain attacks.

Supply chain attacks? But isn’t xz an open source utility?

In this particular case, an individual had used social engineering to very gradually, over multiple years, take over maintenance of the open source project producing xz. Once they had, they slipstreamed in the backdoor in a release they signed.

While many tried to decry this incident as evidence that open source software is inherently insecure (as this sort of social engineering is always a possibility), there’s another side to the coin: it was an open source packager performing standard benchmarking on a development release of an operating system who uncovered the issue. As the adage goes, many eyeballs make all bugs shallow.

One side effect of this attack was renewed interest in Software Bills of Materials (SBOMs). Organizations that are able to produce an SBOM for their software have a record of what they have installed, including the specific versions, as well as what licenses apply. This provides the ability to audit your software — or your vendor’s software — for known security vulnerabilities, and to react to them more quickly. Many organizations are forming DevSecOps teams to manage building, maintaining, and validating SBOMs against vulnerability lists as part of ongoing security in-depth efforts.

Even better, the OSS community is stepping up to build tooling for producing SBOMs into their development chains and utilities. The Node.js community has several projects that will produce SBOMs from application manifests; PHP’s Composer project added these capabilities; Java’s Maven and Gradle each have plugins to generate SBOMs.

Security is and will continue to be a top concern for companies using open source software, and in 2024, we saw proof that the ecosystem is helping protect them. Whether or not we will have another zero-day attack in 2025 remains to be seen, but companies are recognizing the benefit of being more proactive by embedding security best practices into their development and operations workflows and managing OSS inventory with the assistance of tools like SBOMs.

Support Your Entire Open Source Stack

Companies around the world trust OpenLogic to provide expert technical support for the open source technologies in their infrastructure, including LTS for EOL software. Let our enterprise architects tackle the toughest challenges so your developers can focus on what matters to your business.

Explore solutions