Covering Disruptive Technology Powering Business in The Digital Age

Unlike data warehouses, data lakes take on a different approach to the storing of data to a centralised repository. What’s interesting about a data lake is that it allows an organisation to store all of their data, both structured and unstructured, at any scale, as-is without the need for the data to be cleaned, enriched, or transformed beforehand.

That may sound like a recipe for disaster, but in actual fact, data lakes do provide distinct benefits when it comes to analytics. They are flexible and could best address the data realities faced by modern businesses, such as having to make sense of much greater volumes and varieties of data and meeting rising user expectations.

To find out more about data lake analytics and its importance for modern enterprises, DTA interviewed Kitman Cheung, Chief Technology Officer, IBM Data & AI, APAC.

Disruptive Tech Asean: How has data lake analytics evolved in recent years? How does the modern approach differ from legacy approaches?

Kitman:

Data Lake and Hadoop

It’s no secret that knowledge and insights is critical to maintaining a competitive edge over the competitions.  Data lake as an information architecture was created as enterprises’ demand for deeper insights continues to increase.  The original idea is to store raw data in a single repository to satisfy data requirement of the entire enterprise.  An early iteration of data lake is synonymous with Hadoop.  With a distributed computing model and a write-once file system. Hadoop was an appealing technology for building an enterprise data lake, especially back in the mid-2000s.  Since then, technology had marched on and data lake continues to evolve and adopt new technology to keep up with business needs.  Early data lake implementation using Hadoop promised flexibility of structured and unstructured data, schema-on-read flexibility.  Yet, as we have learned since, flexibility without discipline tends to create a large collection of data that is seldom used and hardly useful in driving enterprise business.  While Hadoop promised scalable and affordable platform, it struggled to deliver on some of those promises.  Managing a Hadoop cluster and analytic programming using Hadoop had proven to be laborious.  The MapReduce execution model was must more suitable for batch workload and is often trying for users who may want to explore data quickly to find new insights.  Hadoop started out as a potential replacement for the Enterprise Data Warehouse but ended up augmenting it instead.

Spark and Object Storage

To speed up analytics for data lake, the open-source community came up with Spark.  As an in-memory compute framework, it can process data more quickly and provide much more user-friendly user programming interface to interact with data.  Furthermore, Spark was designed to work stand alone as well as in conjunction with Hadoop. By now, Spark has replaced MapReduce for many big data use cases.  In the last few years, Spark and Cloud Object Storage has emerged as the foundation for a data lake that maintain separation of compute and storage.  Providing faster compute with Spark and more affordable (and more accessible) storage with Object Storage (e.g. Ceph, S3).

State of the Art:  DataOps, Data Virtualisation, Containers…

Up to this point, data lake had primarily focused IT side of the business by addressing issues like the scalability of computing and providing cheap storage.  To really drive business value out of data, the evolution of data lake as architecture has started to shift focus towards aligning with the process and driving the business outcome.  The notion that company can derive value by simply moving data into a single repository has given way to the idea of a flexible data platform that delivers trusted data and quality insights to the organisation in a much shorter time.  This gave rise to the new buzz work – “DataOps”.  Simply put, it is automated, process-driven practised aimed at delivering better data more quickly.  The next generation of Data and AI platform must support more than IT requirements.  It must be designed to support people and processes to deliver a tangible business outcome.  As an industry, we are beginning to realise that metadata management is as important as data management.  You don’t drive business value by moving all data into a cheap repository through ETL.  In fact, the latest trend is focused on building a more agile platform using Data Catalog to improve “findability” and quality of data.  Instead of moving data needlessly, it leverage data virtualisation to create a “virtual” data lake where data stored in different repository appears as a single data source.  This means Data Scientists and Analysts can work on new idea without waiting for new ETL jobs to be build and night ETL batch to complete.  ONLY move the data for operational reasons (such as performance or security requirements).  Finally, we can leverage docker containers along with Spark to create a powerful compute framework that supports multi-tenancy in a compute cluster.

Disruptive Tech Asean: Why should businesses opt for data lake instead of data warehouse? Which is the better option for businesses?

Kitman: I think the problem is a bit more complicated than that.  It is not about choosing a Data Lake vs.a Data Warehouse. I think Data Lake is an overused term that means many things to different people. It’s time to think about a Data and AI platform that encompasses different technologies to provide the best tools for the job.  For structured, relational oriented data set, a data warehouse is still the best choice for analytics and reporting performance. For semi-structured and unstructured data, object storage, NoSQL stores such as MongoDb can be a better choice. To continue to innovate, we need to shift the discussion away from how to store data to how can make data more usable and accessible. The new information architecture needs to start with integrated metadata management across a variety of different data stores.  It should provide seamless access to data regardless of how or where the data is stored.  The platform should use automation and AI technology to augment knowledge workers in every aspect of working with data. Finally, it must build on a flexible and open foundation that allows new technologies in the future.

Disruptive Tech Asean: How can businesses avoid the typical pitfalls associated with data lake analytics so that they can get the most out of their data?

Kitman: Focus on delivering trusted data to the enterprise. It is important to maintain focus on the governance of data.  If someone tells you that one technology or one open-source project will solve all your problems, they are probably wrong.  This is about delivering value to the business and it should not be a “religious” debate on which single technology is the magic bullet.

Disruptive Tech Asean: Can you share with us how your analytics solutions allow businesses to make the most of their data lake?

Kitman: IBM Cloud Pak for Data (CPD) is a comprehensive Data and AI platform that will help our client modernise their existing data lake. Built on an OpenShift platform, it leverages containers to provide a full range of Data and Machine learning services that can run on any infrastructure. Understanding that most enterprises struggled with finding and accessing data, CPD is built around an enterprise information catalogue that manages metadata across the entire organisation. While many of the data / ML services are integrated, the users have full control over what services to use and how many resources they should consume. We have also engaged with business partners such as MongoDB and EnterpriseDB to provide their technology within the platform. Finally, CPD is designed to integrate into our customers’ existing IT/Data landscape. It will integrate and manage data stored in a number of different enterprise data warehouses and Hadoop implementations.

For views on other experts in the field of data lake analytics such as Ritu Jain, Product Marketing Director at Qlik, Geoff Soon, Managing Director for South Asia at Snowflake, Stu Garrow, SVP & General manager APAC, Talend and Jay Jenkins, Head of Customer Engineering for Singapore and Malaysia at Google, you can download our special focus here.

(0)(0)