Covering Disruptive Technology Powering Business in The Digital Age

During his time in Singapore at the Strata+Hadoop Conference, Mike Pickett, who hails from California USA, spoke to Big Community on his thoughts around Big Data integration in the region and the future of Big Data Analytics.

“We see Asia as the fastest growing portion of the market. While it’s coming off a smaller base, that is quickly changing. One of the benefits of being in Asia is that there is very little traditional infrastructure, so a lot of companies are able to move directly into Big Data architectures and technologies because they don’t have to worry about what was going on with legacy technologies”, he says.

To a question as to whether having little to no experience with legacy will be a good approach towards Big Data adoption, Mike feels it is a great approach. He sees Asia as a very exciting region to be in for both the technology vendors such as Talend, as well as Hadoop vendors.

“Coming from the United States and coming from western Europe, it is still a challenge for a lot of companies to make the leap in a big way to these new data architectures, because they have so much to think about in terms of how they move their data over and how they manage their governance. They are doing it, but it is a longer process since there are many more things to think about in terms of legacy infrastructures and regulatory compliance.”

In Asia, which is practically starting from scratch, the implementation is part of the initial project, therefore bypassing and leaping the trial and error curve western countries had to endure.

At the same time, having the supporting skills in place to support the architecture, is still a big obstacle around the world as well as locally.

“Finding the right skill sets is a challenge globally. These new processing frameworks, while they are simplifying things in many ways, they do require advance skills. In this scenario, Talend is able to shrink the amount of time a Data Scientist or data integration developer would need to spend getting their data sources into a big data environment. We assist with processing and changing the data to the way they want to see it processed, using the different data processing frameworks, such as spark or spark streaming”.

An analogy to help understand this process is to take the data as the ‘job’, and getting the data in to that point is the ‘commute’, then the objective is to shorten the commute time to allow people more productive time. Ideally, finding a balance between cost of the required talent and the infrastructure options, the value proposition for a company then, would be in being able to do more work with fewer people, or by making the most with the people they already have.

Mike believes that there are more roles appearing in the market to contend with the ever expanding world of Big Data Analytics. Other than Data Scientists, there has emerged the Data Integration Expert to make the transition towards moving to Big Data a little more palatable.

“Data scientists we often consider as the people who are trying to think of the algorithms. Not always, but often times, that is the classification of Data Scientists. The Data Integration experts are becoming the people that find new data sources, connect to those data sources, bring the data in and change the format of the data so it fits within the environment. Whether its batch or files, regardless the latency. Real time streaming is also becoming a very important tool to consider”.

The key is in creating an easy to use environment for developers to communicate. By having that environment, processing data, changing it, merging it to whatever platform for it to be useful to the company so it can be easily communicated and anyone can come in and look at the work.

“In the beginning, you might have a very advanced developer looking at the initial integrations, and over time, bring in newer developers to extend or work on those original data integration mappings so that you can now get more scale out of your organisation.”

In unconfirmed reports, approximately 60% of a hand coders time is being spent on running legacy management and maintenance. Cutting that time down and bringing more value to those running the hand coding, will drastically improve integration as well as delivery times to the customers. Democratizing data analytics then would become an achievable outcome in the long run, giving autonomy to the people who need instantaneous insights to making decisions.

(0)(0)