Covering Disruptive Technology Powering Business in The Digital Age

As an ever increasing variety of data gets captured and analyzed, we face challenging issues of data governance in general and metadata governance in particular. Data governance has always been a challenge. Organizations have struggled for decades to get well-defined customer or product master data. But now, in the world of data increasing in velocity, variety and volume, they are also faced with maintaining increasingly complex metadata and data lineage caused by not only the variety of ways a piece of information can be used but the increasing importance of relationships among various data elements. This is particularly true in the area of networked devices, the Internet of Things (IoT).

Historically, data governance has been centered on two types of data: Transactional and Master. Implementers of ERP systems have rightly focused on these two types of data.  But, the game changes when instead of just wanting to process transactions (such as orders, payments, payroll) we want to consolidate or analyze transactional information to make better business decisions.

Also, transactions aren’t only coming from ERP systems; in the Big Data world it is also a tweet, a post, a click or a sensor reading. The key is consolidating this disparate information so we are able to make a decision.  This  “Decision Data” results from the combination of information. It could be as simple as finding the mean and variance of orders of a given product or as complex as creating a scoring algorithm for the litigation potential of an email sent to customer service. In short, any time we combine data to create new information we are creating Decision Data and metadata. This metadata needs to be documented and governed.

It is also important to note that in many cases we are dealing with third party data. While the metadata around third party data can be known and trusted, the consumer can’t govern it.

Decision Data has been largely ignored in the field of data governance until fairly recently. The calculations and algorithms often go undocumented. As the number of big data and analytics projects increases, so does the amount of decision data, thus accelerating the problem.

The key point is, if the definition is not well documented or governed and really just captured in the algorithm, then problems will arise. Multiple organizations may duplicate efforts and produce their own definitions, making cross organizational comparisons difficult if not impossible.

Let’s turn our attention to the Internet of Things (IoT), where we have a network of devices of different types producing streams of data. Not only must this data be decoded and transformed, but its interpretation may be dependent to other devices in the same network. Devices of different types, characteristics, and configurations need to be able to be added, which requires a flexible and extensible model.

We need to consider data governance in terms of some new conditions that have resulted from the Big Data movement:

• Data now comes in a wider variety of different forms
• The form of the data and what we intend to do with it determines how we process it

Because of the complex and changing relationships between data elements, the solution for these problems are likely to lie outside the relational world in the area of semantic or graph data technologies.

This article was originally published on www.insidebigdata.com and can be viewed in full

(0)(0)