
Few things in technology outside of catching Pikachu on your iPhone have been hyped as much as big data. However, with any trending topic, new technology or invention that’s being called the “wave of the future,” it’s important to look past the hype and analyze what’s actually going on. Outside of widespread media attention and the increasing importance placed on collecting data, big data is rarely used.
According to MIT Technology Review, only 0.5% of digital data is ever analyzed. From a business perspective, this means that millions of companies are losing significant opportunities to increase efficiency, lower costs, and target new customers because they aren’t using the data they’re collecting.
While the rate of analyzed data is discouraging, new data is being created and collected at an exponential rate. By 2020, approximately 1.7 megabytes of new information will be created every second for every human.
When exploring why big data analysis is lagging and how to fix this problem, it’s important to consider where data is being stored and crunched. While some organizations have been storing their information in the cloud for years, sensitive data is typically kept on-premises. Although secure, building an infrastructure and a massive data processing center to keep up with the amount of gathered data is inconvenient due to the incurred costs and efforts.
DATA ANALYSIS NEEDS COST-EFFICIENT, EASY-TO-IMPLEMENT CLOUD TECHNOLOGY
The cloud has automated some of the heavy lifting in technology today and is cost-effective, but it has not been perfected for big data analytics. Moving large amounts of data in and out of the cloud comes with security risks and performance fluctuations, especially when dealing with terabytes, petabytes, and even exabytes of digital content. Moreover, traditional cloud solutions have yet to meet the bare minimum requirements regarding big data application integration and software orchestration. The tedious jobs of designing, deploying, fine-tuning, and maintaining big data architectures still have to be done by the clients.
Google, for example, has over 105 million active users and collects 14 different types of data including ad clicks, browser information, search queries, and more. Storing and processing this massive amount of data demands a robust and powerful solution that’s uninterrupted and consistent. Such a solution was difficult to achieve with virtualization, where multiple workloads from different companies are running on the same server. A hypervisor, which allows virtualization, also impedes big data performance by limiting processing power when being spread thin among multiple hosts. This is known as the “noisy neighbor” effect, and it has limited the cloud’s potential to faithfully serve big data because a single architecture is serving multiple customers.
Up to this point, the two primary big data solutions are either too costly and time-consuming (on-premises data centers) or unreliable, insufficiently automated and insecure (the virtualized cloud). Without a clear solution, the default has been to do the bare minimum with the available data.
THREE WAVES OF TECHNOLOGICAL INNOVATION
Significant technology innovations, including big data, typically move in sets of three waves. The first wave isinfrastructure, “the cornerstone of big data architecture.”
The second wave includes the tools created to help harness the power of the technology and the third and final wave is applications. With an infrastructure in place and available tools, big data applications are being optimized for cloud use with a blend of technologies such as:
- Programming Frameworks: Hadoop, Apache Spark
- SQL Databases: Oracle, MySQL
- NoSQL Databases: Datastax, Couchbase, MongoDB
- Analytics: Datameer, Platfora, Trifacta
- Visual Analytics: Tableau, Zoomdata
The cloud is getting lighter for big data. The bare-metal cloud, also described as a dedicated infrastructure, is brightening the future of cloud-based big data projects. While the traditional cloud has performance bottlenecks and security risks, a bare-metal or dedicated infrastructure removes the uncertainty, providing predictable performance and single-tenant isolation. This eliminates the noisy neighbors and allows companies to power their big data efforts with dedicated hardware.
With uncertainty removed, costs minimized and security ensured, the cloud is resurfacing as a viable big data solution. Moreover, next-generation big data clouds offer automation and orchestration at every layer in the technology stack, starting with the underlying bare-metal infrastructure and spanning everything from application configuration and tuning to dependency and software upgrade management. The time has come for big data architects to take another look at the cloud as the primary facilitator of big data, empowering companies to affordably analyze information faster and on a greater scale.
This article was originally published on dataconomy.com can be viewed in full


Archive
- October 2024(44)
- September 2024(94)
- August 2024(100)
- July 2024(99)
- June 2024(126)
- May 2024(155)
- April 2024(123)
- March 2024(112)
- February 2024(109)
- January 2024(95)
- December 2023(56)
- November 2023(86)
- October 2023(97)
- September 2023(89)
- August 2023(101)
- July 2023(104)
- June 2023(113)
- May 2023(103)
- April 2023(93)
- March 2023(129)
- February 2023(77)
- January 2023(91)
- December 2022(90)
- November 2022(125)
- October 2022(117)
- September 2022(137)
- August 2022(119)
- July 2022(99)
- June 2022(128)
- May 2022(112)
- April 2022(108)
- March 2022(121)
- February 2022(93)
- January 2022(110)
- December 2021(92)
- November 2021(107)
- October 2021(101)
- September 2021(81)
- August 2021(74)
- July 2021(78)
- June 2021(92)
- May 2021(67)
- April 2021(79)
- March 2021(79)
- February 2021(58)
- January 2021(55)
- December 2020(56)
- November 2020(59)
- October 2020(78)
- September 2020(72)
- August 2020(64)
- July 2020(71)
- June 2020(74)
- May 2020(50)
- April 2020(71)
- March 2020(71)
- February 2020(58)
- January 2020(62)
- December 2019(57)
- November 2019(64)
- October 2019(25)
- September 2019(24)
- August 2019(14)
- July 2019(23)
- June 2019(54)
- May 2019(82)
- April 2019(76)
- March 2019(71)
- February 2019(67)
- January 2019(75)
- December 2018(44)
- November 2018(47)
- October 2018(74)
- September 2018(54)
- August 2018(61)
- July 2018(72)
- June 2018(62)
- May 2018(62)
- April 2018(73)
- March 2018(76)
- February 2018(8)
- January 2018(7)
- December 2017(6)
- November 2017(8)
- October 2017(3)
- September 2017(4)
- August 2017(4)
- July 2017(2)
- June 2017(5)
- May 2017(6)
- April 2017(11)
- March 2017(8)
- February 2017(16)
- January 2017(10)
- December 2016(12)
- November 2016(20)
- October 2016(7)
- September 2016(102)
- August 2016(168)
- July 2016(141)
- June 2016(149)
- May 2016(117)
- April 2016(59)
- March 2016(85)
- February 2016(153)
- December 2015(150)