Big data equals big opportunities

Every two years the volume of data generated across the world more than doubles. To put it another way, 90 per cent of the data ever recorded was laid down in the last two years. As more and more devices and everyday objects are fitted with their own sensors and communications, ever more detailed information is made available for analysis. And as the power of analytic tools increases, costs are reducing. The resultant ability to identify patterns provides quicker, more accurate answers to all manner of questions and, indeed, makes it possible to answer questions that have never even been asked before.

Several enabling technologies underlie the growth of “big data” analytics. Sensor, processing and communications technologies make it possible to construct smart objects, and enable them to capture enormous quantities of data. Then there is the increasing speed and power of hardware: both processors and memory continue to follow Moore’s law – doubling in power (or halving in relative price) every 18 months to two years. Techniques such as in-memory processing are offering even greater advances in processing power for massive databases. Software such as the open-source Hadoop makes it possible to execute large scale processing of data sets on clusters of commodity computers, and this in turn reduces or eliminates one of the barriers to entry into advanced analytics and allows small and medium enterprises to join the game.

Mark Beyer of the technology research firm Gartner, quoted in the Formtek blog, sees three principal commercial opportunities for what he describes as “Extreme Information Management”. The analysis of social data will reveal network relationships and sentiments that can be used to improve marketing processes. Monitoring operations and collecting better data and metrics will be used to enhance logistics and supply planning. Finally high-resolution data seta will be used in the search for competitive advantage, by fine-tuning price and performance.

The applications for data analytics are almost endless. Current examples include the analysis by streaming video service providers of real-time traffic patterns across different localities and device types, which enables them to drive improved reliability and determine where extra capacity will be needed. Other web-based service providers analyse the information that they hold on individual users to identify individual optimum price points.

Away from the web, analysis of footfall and traffic past billboards is correlated with buying behaviour, leading to a better calibrated determination of advertising fees. In the field of medicine, our understanding of complex conditions such as heart disease is being improved by the detailed analysis of factors such as diabetes, associated medications, inherited conditions and lifestyle. Another application is analysing the performance of experts such as doctors or lawyers when compared with their peers, and relating this to aspects of their professional behaviour and practice in order to generate continuous incremental improvements.

It is difficult to comprehend even the current scale of date creation and capture, never mind the future. Cisco’s Global Information Technology Report 2014 gives 2.5 quintillion bytes per day as the current rate of data creation, but puts this into perspective by pointing out that one hour of customer transaction at Walmart generates 167 times the data stored by the US Library of Congress.

Despite the volume of data available, and the increasing availability of powerful processing tools, it is widely observed that there is currently a substantial gap between data volume and data value; but there is general agreement that this gap will be closed within the next decade. Few sources are confident to estimate the consequent scale savings or revenue generation, but big data is widely described as a “trillion dollar opportunity”.

The exploitation of big data is not without its challenges however. The technical issues of storage capacity and processing power are perhaps the easiest to overcome. A set of more difficult problems concerns the way in which people interact with the data. A growing cohort of data scientists provides a partial response, but such experts are in short supply and it is difficult to grow this resource quickly enough. Another approach is the development of disciplines such as visual analytics which put the ability to manipulate data and extract meaning in the hands of the wider knowledge management workforce.

For a complete picture, many analytic initiatives require data capture from across the world, including locations well beyond the reach of terrestrial telecommunications networks. A ready solution lies in the use of mobile satellite communications services.

For smartphone, tablet or laptop based data capture, Wi-Fi enabled portable data satellite terminals such as BGAN provide a highly portable, mobile and cost effective solution. Inmarsat’s suite of machine-to-machine services provides an alternative that is tailored to the management and monitoring of remote assets across the globe. In many instances a “big data” business case that does not add up when restricted to an urban or local view typical of the terrestrial broadband communications footprint suddenly makes sense when the power of global satellite communications is harnessed.

So when considering the feasibility of a big data initiative, be sure to understand the potential for satellite-enabled M2M communications to push the cost-benefit analysis beyond the tipping point.

*In his next blog, Ted Flint will examine the Cloud and anything-as-a-service.