Covering Disruptive Technology Powering Business in The Digital Age

Dan Ariely, professor at Duke University says, “Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.” Without doubt, big data is almost like a fashion statement. Every engineer wants these two words on his or her resume.

Organisations want to claim that their strategies are driven by data—that too, the new, fashionable big data. However, there’s a gap in what organisations desire, and what they are able to achieve. Companies find it embarrassing to admit that they are unable to fully tap into big data and that they are failing to derive worthwhile insights from the corpus of data that they have collected over the years. But what’s interesting to understand is: why this embarrassing gap?

What insights are you looking for?

From where we stand right now, it wouldn’t be wrong to say that some discovery from big data happens serendipitously. Data sciences teams in such organisations employ a bottom-up approach. They try out some semi-informed trial-and-error correlation to derive interesting patterns. Sometimes it does surface some very interesting insights. However, organisations don’t like to depend on serendipity alone. It doesn’t scale. And there is always a fear of deriving too much confusion from big data, as said rightly: data would confess anything if you torture it enough!

So what is the next step? In order to get meaningful insights from data, there needs to be some method to the madness. Instead of a bottom-up approach, organisations need to drive a top down approach especially if there’s a sea of data that one is dealing with. There needs to be a clear understanding of the questions that an organisation is seeking answers for. What impact does that insight have on the overall business? If you had that insight into your business, how would that information help you shape your strategy or tactics? The answer-seeking inquiry has to be deliberate, and focused. Defining the problem is half the solution—and this is the most difficult part. Most organisations that are frustrated with the ROI from their big data initiatives actually fail at specifying the questions that their big data can provide answers for.

Once you are done with the problem statement, the rest of the flow is easy—well, almost. You know what data to gather and analyse. You know what kind of teams to build and retain. Your engineering teams would figure out the tools and technologies, and your product-management and product-strategy teams would lay down the metrics that they would like to monitor and track.

Do you have the right kind of data?

Data could mean anything, starting from booking details, to a customer’s demographics, to time-and-location specific tracking of consenting users. Since we are talking about big data, we assume that you already have a huge corpus of data that you want to start with. Let’s also assume that you have a capable team that can sift through the big data and derive insights from it. So, what’s the problem? What could potentially derail your big data plans?

Well, there are three categories of problems with data alone:

Noisy data: When you have a lot of data, most of it is irrelevant. This could be a typical problem since one tends to spend more time and effort filtering out the noise. Unstructured data has more noise than structured data, and the data-capture teams would typically devise ways of imposing a structure on the data that one collects.Cold start: This situation is when you have too little data collected. When people start, they almost start out this way—especially the new-age, consumer-internet companies. This is true of traditional offline businesses that have just about started to collect digital data. Is this really a problem, then?

Depends—mostly on how much data is trickling in on a daily basis. If the rate of data-collection is good, you’ll eventually build up a respectable corpus and your big-data teams can, then, do some interesting stuff with the data.

Missing key attributes: You have the right volume of data, but some of the critical attributes that should have been present in the data are actually missing. For instance, you want to gain insight on behaviour of logged-in customers…and some of the critical flows in the user-interaction journey forget to capture the user-id. This could potentially put you back in the cold-start bucket.

Who’s going to do it?

This is probably the trickiest part given the current market dynamics. While the reality catches up to the hype, organisations are in an all-out talent-war. The hype around big data complicates the dynamics even further. A good portion of the resumes that you would collect would have all the right keywords, but not enough substance to back up the claims.

Assuming that you are successful in hiring some good data-sciences experts, where do you put them in the organisation? Would that be a horizontal centre of excellence, or would the data-engineers be distributed in the vertical lines of businesses? There are success stories— as well as horror stories—on both sides. To establish early success stories, you certainly want to assign some straightforward use-cases to the data-team, which also get integrated within the user- experiences and impact-assessment within the lines of businesses. This helps getting executive sponsorship for continued investment in big-data. At the same time, the big data engineers see where they fit and how their work creates a long-lasting impact for their organisation.

This article was originally published on www.financialexpress.com and can be viewed in full

(0)(0)