As the Big Data landscape has changed, comparing Apache Hadoop vs. Cloudera and their commercial platform is a worthwhile exercise. Do enterprise teams still need Cloudera for their Big Data stack management or can they save by independently managing their Apache Hadoop implementation?

In this blog, we’ll take a close look at the value of the Cloudera platform’s software bundle, proprietary tools, and cloud-hosting services. We’ll also explore Cloudera alternativesfor organizations that would prefer to not migrate to the cloud and want the freedom to decide where and how to manage their data infrastructure. 

Note: In this blog, references to the Cloudera platform are meant to encompass both the Cloudera Data Platform (CDP) and the legacy product, Cloudera Distribution of Hadoop (CDH).

Apache Hadoop vs. Cloudera: What’s the Difference?

Apache Hadoop is a free, open source data-processing technology that uses a network of computers to solve large data computation via the MapReduce programming model. Cloudera offers a commercial, Hadoop-based platform that is available via paid subscription.

The Cloudera platform is based on Apache Hadoop and various other software packages that, by and large, are part of the broader Apache Hadoop ecosystem. Therefore, many of the features and functions of Cloudera’s platform are available for free via the collection of those foundational open source software packages. 

When customers pay for a Cloudera subscription, they are essentially paying for:

• A curated bundle of the open source software packages and specific versions that have been validated and proven to work together.
• A couple of proprietary (not open source) applications that provide conveniences intended to help adopters manage an implementation of these disparate open source software packages.
• A hosted managed services provider that unites it all in a controlled environment with the promise of stability, availability, and carefree maintenance.

While valuable for some enterprise use cases, these benefits come at a price — particularly the last one, as cloud migrations can be expensive. Because the Big Data landscape is continuously evolving with new solutions coming on the market all the time, it is a good practice to regularly evaluate the return on investment of those features against the cost of managing an equivalent open source stack. 

In the next few sections, we’ll dig deeper into the three bullets mentioned above and compare them to the free equivalents in Apache Hadoop.

Back to top

1. Cloudera’s Curated Bundle of OSS

When the Hadoop Ecosystem was an emerging technology, it was beneficial to have a leader in the space like Cloudera piecing together and testing a set of immature open source technologies that were under active development. Cloudera made it so individual companies did not have to dedicate development resources to keep pace with many independently evolving software releases and ensure there were no breaking changes at all the integration points. This can be particularly painful for early adopters, as there are rarely standards or best practices in place to allow product features to evolve independently. Without standards, the products are more tightly coupled and implementations must be more closely managed. 

The situation today, however, is very different. For example, many products now rely on JSON or YAML as the agreed-upon data exchange formats, but those were not in place 20 years ago. Data formats like Parquet and Avro take this a step further. Likewise, there are best practices around RESTful API versioning that many products now implement — and the list goes on. So what would have been very burdensome and resource-draining when Hadoop first emerged is considerably more feasible these days because standards and best practices have caught up. 

This is not to say a controlled and validated environment isn’t a good thing. It just might not deliver as much ROI for organizations as it once did. Furthermore, one must reevaluate being locked into a bundle vs. having flexibility now that more innovative and impactful technologies are available. Specifically, there are a couple of foundational areas where Apache Hadoop has made considerable advancements compared to what you get with the Cloudera implementation of Hadoop, and that’s what we will cover next. 

Execution Services: Oozie vs. Airflow

At a time when more modern organizations are moving toward Apache Airflow for workflow, Cloudera is still shipping with, and relying on, Apache Oozie. Apache Oozie workflows are tied to the Hadoop ecosystem and require unwieldy XML-based definitions. In contrast, Apache Airflow is a more modern, flexible, and scalable workflow and data pipeline management tool that integrates well with cloud services and various systems beyond Hadoop. It has a friendly user interface, a strong community, and advanced error handling. 

Security Services: Navigator & Sentry vs. Atlas & Ranger 

Modern Apache Hadoop implementations use a combination of Apache Atlas and Apache Ranger. Both of these products achieve significant improvements over the legacy Navigator and Sentry. Atlas will be covered again later when highlighting data governance. Apache Ranger has a more user-friendly web-based interface that makes it easier to create and manage security policies. Unlike Sentry, Ranger includes built-in robust auditing capabilities for tracking events and activities across the platform, even outside of Hadoop proper.

To be fair, Cloudera is migrating to these improved options as well, but they are not there yet — leaving CDP implementers saddled with the complexity of a combined solution but unable to benefit from the full set of new features.

Back to top

2. Cloudera’s Proprietary Tools for Cluster Management, Cluster Administration, and Data Governance

Cloudera ships two proprietary applications, Cloudera Manager and Cloudera Navigator, to provide implementors with a toolkit for managing and administering their Hadoop Cluster. These applications are essential in offering a cohesive, professional, and useful Hadoop-based Big Data platform. 

However, there are open source alternatives that meet or beat the features available in these proprietary tools. In fact, the most predominant open source versions of these tools were originally developed in the open and handed over to the Apache Foundation by Hortonworks — a company that was purchased by Cloudera in 2019. 

Cloudera Manager vs. Ambari

Cloudera Manager is an administrative application for the Cloudera Data Platform (CDP). It has a web-based user interface and a programmatic API, and is used to provision, configure, manage, and monitor CDP-based Hadoop clusters and associated services.

Apache Hadoop implementors use Apache Ambari (a project with Hortonworks origins) to accomplish what is offered through Cloudera Manager on CDP Hadoop implementations. Apache Ambari has a web-based user interface and a programmatic REST API that allows organizations to provision, manage, and administer Hadoop clusters and associated services.

To take a deeper dive and learn more about the nuanced differences between these tools, see my previous blog: Apache Ambari vs Cloudera Manager. 

Cloudera Navigator vs. Apache Atlas

Cloudera Navigator handles data governance. It offers a wide range of features for auditing and compliance, from organization policy creation and tracking to regulatory requirements like GDPR and HIPPA. It also includes data lineage tracking to look back upon data transformation and evolution, as well as metadata management for tagging and categorizing data to assist in searching and filtering.

Apache Hadoop implementors use Apache Atlas (also originally developed by Hortonworks) to implement data governance and metadata management. Cloudera Navigator is only applicable to CDP, whereas Apache Atlas works across a broad range of Hadoop distributions and data ecosystems. It is extensible and integrates with other packages, like Apache Hive and Apache HBase.

Apache Atlas logs creation, modification, access, and lineage information about each data asset. It tracks who has accessed or modified data to provide an audit trail for compliance and monitoring purposes. Policies can be defined in Atlas to manage role-based access control (RBAC), attribute-based access control (ABAC), and data masking. To enforce these policies, Atlas integrates with Apache Ranger (another open source package in the Hadoop ecosystem).

Back to top

3. Cloudera’s Cloud-Hosting Environment and Managed Services

Measuring the value of where the infrastructure resides will likely be more of a policy question for most organizations. Most organizations have a preference or a requirement that dictates whether they host services in public, private, on-premises, or hybrid clouds. So the real assessment here lies more in the value aligned with the managed services offered by Cloudera. For organizations that are not required to manage and own their own infrastructure, and don’t mind paying for these managed services, this may tip the scales in Cloudera’s favor. 

However, organizations that don’t want to be forced to the cloud should consider whether they have the talent, motivation, and capacity to own and maintain an Apache Hadoop implementation. The maturity of the Hadoop ecosystem and the availability of standardized cloud resources make this a viable alternative to Cloudera — but only if you have the internal resources or a partner like OpenLogic with deep Apache Hadoop expertise.

Back to top

Other Considerations 

We outlined some key differences in cluster execution services, cluster security, cluster administration, and data governance between Apache Hadoop and CDP. However, there are a number of other features and functions that are nearly identical for both of these platforms that will require installation, configuration, care, and feeding. These include products like Zookeeper for cluster coordination, and a number of data services that can be applied to meet various needs of an organization. These include, but are not limited to, HDFS, MapReduce, Yarn, Apache Spark, Apache Kafka, HBase, Hive, and Hue.

Back to top

Final Thoughts

There was a time when it was easier to associate a clear value for the dollar spend on Cloudera. They were pioneers in Big Data and offered the first commercial bundle of Hadoop. They were the Hadoop provider for many of the Fortune 500 firms. The Cloudera Platform could speed time to market, providing a clear path to a stable Big Data environment that allowed implementers to focus on creating domain-specific applications that leveraged their data — rather than juggling between managing a data platform and making use of their data.

However, nearly two decades have passed since the first incarnation of Hadoop. Cloudera has been involved for over 15 years, and a lot has changed. Hadoop has matured dramatically, and the supporting ecosystem has grown. New open source solutions are being developed all the time, as well as new commercial offerings around Big Data services and support. While there is still an appetite for hands-off, fully managed Big Data platforms like the one that Cloudera offers, the price has driven demand for lower-cost alternatives. For some organizations, using Apache Hadoop and avoiding a costly cloud migration is priceless.  

Information about CentOS Stream 10 has been trickling in since ISOs first became available in June. CentOS Stream 10 will be based on Fedora 40 and released sometime ahead of RHEL 10, but the current images are still in testing/development and could very well change between now and the actual release. 

So what do we know about CentOS Stream 10? Our expert weighs in and offers considerations for enterprise teams considering CentOS Stream for production workloads.

CentOS Stream Project Update 

CentOS Stream has an interesting history, with some notable developments in the past few years. After announcing in 2020 that CentOS Linux would be discontinued in favor of focusing on CentOS Stream, last year Red Hat ruffled more feathers by announcing that CentOS Stream would become the sole repository for RHEL source code. CentOS Stream 8, the first release, reached end of life in May 2024; CentOS Stream 9 has been out since 2021. 

On June 6, 2024, the CentOS Project posted links to the CentOS Stream 10 compose images, install ISOs, and container images with the following message: “Please note the compose is still taking shape. Packages are still being added and even removed at this point. Not all packages are fully onboarded to gating, so just some updates are landing. Packages are being moved between repositories. Comps groups are being updated…” Developers were encouraged to test and share feedback.

In other words, much is still to be determined. New ISOs have been made available periodically since the June announcement (as of this writing, the last batch dropped on October 22, 2024). 

Back to top

CentOS Stream vs. CentOS Linux

The main difference between CentOS Stream and CentOS Linux is that CentOS Stream is upstream of RHEL, with packages planned for upcoming releases, and CentOS Linux is a rebuild of the current RHEL release.

Another key difference is how updates are made in the two distributions. For CentOS Linux, new minor versions consist of large batches of updates, with smaller updates between versions. Rather than batch updates, packages in CentOS Stream are updated as they are ready, in a continuous stream, and there are no minor versions. 

Before all versions reached end of life, CentOS Linux had a community support lifecycle of ten years, like RHEL and many other Enterprise Linux distributions. CentOS Stream has a shorter lifecycle of five years, with EOL based on when the corresponding RHEL release leaves Full Support and enters its Maintenance Phase (security updates only). 

Back to top

How Long Will CentOS Stream 9 Be Supported?

CentOS Stream 9 will be supported until May 31, 2027, when RHEL 9 leaves Full Support.  

Back to top

CentOS Stream 10 Release Date

CentOS Stream is upstream of RHEL and all signs point to the RHEL 10 GA release sometime in the first half of 2025, so the CentOS Stream 10 release is anticipated in late 2024 or early 2025. 

Back to top

Notable Changes in CentOS Stream 10 

• Kernel: CentOS Stream 10 will be using a 6.11-based kernel, rather than 5.14 that CentOS Stream 9 used.
• Programming language support/compilers: CentOS Stream 10 has GCC 14.2.1 (instead of GCC 11.5), and Python 3.12 (instead of Python 3.9).
• CPU compatibility and capabilities: one user encountered a warning message that that x86_64-v3 will be required at a minimum in the future, but as of now it is just a deprecation warning.
• Performance: Phoronix ran some benchmarks, and a thorough comparison of performance is available here. That is for Arm64 instead of x86_64, but should still be comparable.

Back to top

Using CentOS Stream in Production

There is some debate over whether enterprises should use CentOS Stream in production. Some say the rolling release model makes it too unstable and that it’s more of a ” beta testing ground” for features, or a preview of the next version of RHEL (though not everything in Stream may make it into RHEL). Red Hat explicitly says that CentOS Stream “is not designed for production use in enterprise environments” and recommends using RHEL as a CentOS alternative.

However, depending on your use case, using CentOS Stream for production workloads may not present any issues. Some teams like that Stream gives them access to bug fixes and new features before they become available in RHEL. The notion that CentOS Stream is fundamentally less stable or reliable than RHEL is not really accurate, as everything in Stream undergoes QA and testing, and has been accepted for the next minor RHEL release before being merged into Stream.  

The main difference between RHEL and CentOS Stream comes down to commercial support and services that RHEL provides to its paying subscribers.  

Still, a lot depends on your particular use case and infrastructure to determine whether or not CentOS Stream is the right fit. 

Back to top

CentOS Stream 10 Migration and Upgrade Considerations

As usual, you will want to test thoroughly before upgrading important systems. The new kernel version may not support older hardware, and with x86_64-v3 coming in the future, some older hardware may not work at all. Information about glibc-hwcaps can be found here. RHEL 9 did the same with x86_64-v2 and a simple test under Proxmox using x86-64-v2-AES produced a kernel panic during just an install, but x86-64-v3 succeeded.

With a new kernel, glibc, gcc, Python, and other changes, some existing software may not have library versions available to run the older version. Containers or VMs could mitigate the problem, however.

Back to top

What to Expect from Future CentOS Stream Releases

In future CentOS Stream releases, you can expect continuous upgrades of packages, with new versions, security patches, and performance improvements. Future releases may introduce new features, such as updated kernels, newer versions of programming languages, and support for emerging hardware or software trends.

Back to top

Final Thoughts 

CentOS Stream 10 gives us insight into what is likely to be included in the next version of RHEL — the first major release in four years. As to whether CentOS Stream 10 is a viable alternative to CentOS Linux or the best Linux distro for your organization, I recommend checking out this CentOS Stream checklist for guidance. 

It’s always a good idea to have technical support for your mission-critical workloads, and ideally, to work with experts who have full stack expertise to troubleshoot issues with updates and integrations. If you decide to use a FOSS Linux OS, it’s wise to pair it with commercial support from OpenLogic so you always have immediate access to Enterprise Architects. 

Apache Kafka issues, especially for enterprises running Kafka at scale, can escalate quickly and bring operations to a halt. The open source community may be able to offer assistance, but in some situations, you need a resolution fast. 

While some organizations partner with OpenLogic for ongoing, SLA-backed Kafka support, our Professional Services team gets involved when a customer who does not have a support contract needs a consultation or help troubleshooting an issue with their Kafka deployments. These engagements can last anywhere from a few days to a few weeks, depending on the scope and complexity of the project. 

In this blog, we present four Kafka case studies with details on what the Kafka issue was and how OpenLogic solved it. 

Case Study #1: Large Internet Marketing Firm

Background: This customer was tracking clickstream events to measure ad campaign success. Their large bare metal implementation contained 48 nodes, and was processing roughly 5.8 million messages per second with 1-2 second end-to-end latency.

The Issue: LeaderAndIsr requests were failing during rolling restarts, resulting in multiple leader epochs with stale zkVersions.

The Solution: OpenLogic identified an existing bug that had not been fixed in the version of Kafka they were using, which had a higher likelihood of occurring during resource contention on the Zookeeper instance co-located on five of the Kafka nodes. They recommended upgrading the Kafka cluster and running Kafka on Zookeeper on independent nodes, which fixed the issue. 

Length of Engagement: 5 days 

Case Study #2: Large South American Bank

Background: This customer was currently utilizing IBM MQ and not hitting the performance metrics they desired. They were having to deal with large messages at high volume.

The Issue: Due to slow response times with end-to-end latency and total throughput with large messages, the customer wanted to move to Kafka to have a streaming-focused messaging bus.

The Solution: OpenLogic provided architecture using the Saga pattern with Apache Kafka and Apache Camel for managing long-running actions, such as crediting a payment on a loan from cash deposited at a branch. They also provided architecture for using Kafka with log shipping and the ELK stack, as well as for bridging events from IBM API Connect Cloud to Elasticsearch index behind the firewall using Apache Kafka. Finally, OpenLogic led a 5-day Apache Camel training to a team of 15 people so they could learn how to create Kafka consumers and producers.

Length of Engagement: 27 days 

Related Video: Apache Kafka Best Practices 

 

Case Study #3: U.S. Aerospace Firm

Background: Originally this customer wanted help with Rancher and moving from a VM-based Kafka cluster. They were utilizing a web socket server that was responsible for collecting satellite location data in real time. The web socket server could not talk directly with Kafka, and so they had developed a Camel-based system for their original Kafka cluster. They did not have any metrics collected on the existing cluster and could not identify the root cause for message delays and lag. 

The Issue: Performance issues with pub/sub relay application that consumed from websockets from domain-specific appliance and published to Kafka queues.

The Solution: OpenLogic implemented Rancher clusters dedicated to running the Strimzi operator and serving Kafka clusters. They were also able to improve throughput dramatically by moving existing Java code to Apache Camel with vertx driver. 

OpenLogic created metrics with Prometheus and Grafana in both the Camel websocket relay application and the Kafka brokers to determine replication and processing lag, and put monitoring in place to alert on topics that didn’t meet SLAs. Once metrics collection with Grafana and Prometheus were put in place, existing bottlenecks became identifiable and addressing them drastically improved end-to-end performance.

Length of Engagement: 3 days 

Case Study #4: Global Financial Services Company

Background: Customer came to OpenLogic with a security concern with Kafka Connect that violated PCI compliance as well as internal security standards.

The Issue: Sensitive information was included in stack traces with Kafka Connect.

The Solution: OpenLogic created a test harness, which was sanitized so that customer information was not present, that reproduced the bug. They filed a bug against the project and attached the test harness – and wrote the code that resolved the bug. OpenLogic then submitted the code to the community and worked with community to modify the PR to meet the community’s standards. Finally, they informed the customer when the bug was accepted and estimated which release was likely to include the fix for it. As a result, this K.I.P. was produced from the engagement.

Length of Engagement: 20 days 

Final Thoughts

Apache Kafka is an extremely powerful event streaming platform, but when things go wrong, they go wrong at scale. These Kafka case studies illustrate the benefits of having direct access to Enterprise Architects with deep Kafka expertise in those moments when every minute counts.