Big Data Is A Big Deal

When I began writing this column there were key trends in business and technology such as cloud computing, mobile devices and social media which provided a wealth of opportunities for discussion. Many of the columns over the last few months have included information, observations and my perspectives on these topics. Another major trend which has not occupied much space here on the bridge is so-called "big data."

This is a term, like the cloud, which can mean a lot of different things to different people. Let me see if I can shed some light on what we mean by big data and how it will, sooner or later, impact you and your business.

IT professionals have been collecting and finding ways of organizing data to help run companies since the dawn of data processing. Database technologies were devised and perfected through the years to capture ever more efficiently all of the transaction level data needed to adequately operate and control the business. Until fairly recently, the drive has always been to construct the "one truth" or central source for data which was collected from all parts of the company and rationalized, adjusted, corrected and published back for analysis and reporting.

When the central database became to large or unwieldy, the data mart was born. These extracts from the mother ship provided smaller, more focused data sets. Specific deprtments such as sales, marketing or finance would filter out only those data elements needed for a particular analysis or report. This also removed significant burden from underlying systems sensitive to the time needed to process transactions.

The advent of the internet, connected devices and social networking have lead to a data explosion. For a while, efforts were made to consolidate these external sources into the central database in the same manner as internal financial, production and other types of data. Faster database engines, new software and appliances have come along, but the data growth continues to outpace them.

The sheer amount of data can often be more than traditional database technologies can handle. For example, an oil rig may generate 25,000 operational  data points per minute. Trying to store and process this in conventional ways is nearly impossible.

Moreover, the preponderance of data extracted from social networks and other external sources is unstructured and, by its very nature, incapable of being handled by conventional database technologies.

Big data is the effort under way for companies to get their arms around these new sources of information and to try, using new technologies and innovative techniques, to derive some value from it.

We'll talk more about some specific examples of companies listening the the "voice of their customers" by monitoring social networks, and how they can shift from being reactive to proactive in the process.

Think about the new sources of data that may be important to your business and ask yourself are you ready to deal with it?

Captain Joe

Follow me on Twitter @JPuglisiLLC