Covering Disruptive Technology Powering Business in The Digital Age

Being the second iteration of the Apache Hadoop framework, Hadoop 2 is positioned for a wider use in BDA. It runs event processing, streaming, real-time and operational applications too. It’s capable of supporting programming other than MapReduce as well as a variety of other analytical applications.

How Hadoop 2 works is by adding support for running non-batch applications through YARN. Short for Yet Another Resource Negotiator, YARN is a redesigned cluster resource manager that eliminates Hadoop’s sole reliance on the MapReduce programming model.

Hadoop 2 can run a variety of applications since YARN puts resource management and job scheduling functions in a separate layer beneath the data processing one and won’t need to rely solely on MapReduce.

Apart from that, Hadoop 2 is able to improve availability and scalability with an Hadoop Distributed File System(HDFS) high-availability (HA) making available a new NameNode architechture to Hadoop. Its scalable up to 10000 nodes per cluster compared to only 4000 nodes limit in Hadoop.

Where Hadoop clusters used to have one NameNode with a directory of HDFS files, the Hadoop 2 HA allows clusters with redundant NameNodes and therefore eliminating the possibility of a lone NameNode becoming a single point of failure in the cluster. Plus with the HDFS federation capability, it allows clusters to be built horizontally using multiple NameNodes working independently and sharing the same data storage pool, offering a much better compute scaling option.

Back-up and disaster recovery support for Microsoft Windows is available through a snapshot capability with read-only point-in-time copies of a file system. Hadoop 2 also offers binary compatibility for existing MapReduce applications on Hadoop.

Its not all smooth sailing however. Running MapR API requires additional files to execute in Hadoop 2 for any program written in Hadoop, whereas MapR API is easily compatible with Hadoop.

Another significant weakness with Hadoop 2 is with privacy and security of personal data.

Merv Adrian, analyst with Stamford, Conn. based Gartner Inc was quoted as saying, “So much of big data is about combining multiple data streams and getting a wider, broader view of customers, that people are becoming increasingly concerned about how to ensure the privacy of data.”

(0)(0)