Published/updated: October 2012
By Tony Lock
Anyone currently employed in any area of the IT business will be aware, however reluctantly, of the considerable amount of effort being put into marketing ‘Big Data’. During August and September of 2012 Freeform Dynamics surveyed 502 readers of The Register (for sample breakdown see here) to gauge how far organisations have advanced in their adoption of Big Data solutions and the current state of play concerning business analytics. This article looks at where organisations report they currently hold business data, how well they exploit the value locked in their data stores and their thoughts on how things may develop in the near future.
Today the words ‘Big Data’ are currently used and abused in a myriad of ways. From these it is possible to distil the term to be shorthand to refer to a number of advanced data storage, access and analytics technologies aimed at handling high volume and/or fast moving data in a variety of scenarios. These typically involve low signal-to-noise ratios, including but not limited to brand monitoring, log file analysis, high volume transaction monitoring for fraud detection, to name just a few.
Data volumes are growing
It is no news that the amount of data being generated by organisations continues to grow rapidly and this survey illustrates just how much of a challenge this is with over half of respondents reporting that they are experiencing high, or extremely high, rates of data growth and very few experiencing no growth at all (Figure 1).
Of considerably more importance with the vendor focus firmly directed at “Big Data”, is the fact that whilst the expansion of volumes associated with unstructured data sources such as email, office productivity documents, video files etc. is very strong, the reports of growth in structured data sources, such as those associated with relational database systems, is not that much lower.
This begs the question, just how valuable or important are the types of information held in structured sources compared with unstructured repositories?
The value of data
It is evident that the majority of organisations today still keep most of their business critical information in structured repositories, typically relational database management systems (RDBMS). Far fewer hold such data in unstructured sources (Figure 2).
That said, it is worth noting that nearly one in ten of the respondents indicate that they are not sure how such data is held. It is easy to understand why this should be so when the question of data storage architecture has never been an area of intense study for most IT professionals and has never even been thought about by the vast majority of business users.
When asked how things are changing, the research indicates that roughly equal numbers of respondents expect there to be a shift towards either structured or unstructured sources as the repositories for business critical data. That said, the most frequently offered opinion is that there is likely to be little change evident in the near future. This illustrates that Big Data will not be sweeping away the very strong position that relational databases and other established solutions hold in the near term. With so much data out there just how well are organisations exploiting what they already store?
Data value is still ‘underexploited’
The Reg readers who responded to the survey tell us that only rarely is the data their organisations hold exploited as well as it could be. Even a cursory glance at Figure 3 shows very clearly that only around one in eight organisations fully exploit their structured data whilst an even smaller fraction does so for unstructured data.
By Tony Lock
A recent global survey of 1279 IT and business professionals highlighted that rapidly changing business and regulatory demands are driving a need to modify how security is managed in their software development processes. ...more
By Dale Vile
In the drive towards ever faster and more granular software delivery cycles, it’s important to ensure that speed and responsiveness don’t come at the expense of quality. Insights from 327 IT professionals in a recent survey shed light on the issues and practicalities. ...more
By Richard Edwards
By Dale Vile
By Bryan Betts and Dale Vile
Yesterdays software delivery processes are not up to dealing with today’s demands, but modernising you approach is not just about implementing Agile, even creating a DevOps culture. You need to focus on some specific, hard-core principles. ...more
By Dale Vile & Jack Vile
Cloud services are increasingly becoming part of the IT delivery mix, but a recent study of 378 senior IT professionals suggests a parallel commitment to ongoing investment in the datacentre. This in turn shines a light on the key role of modern application platforms. ...more