Platt Perspective on Business and Technology

Big data and the assembly of global insight out of small scale, local and micro-local data 9: rethinking potential operational strengths and weaknesses

Posted in business and convergent technologies, reexamining the fundamentals by Timothy Platt on January 22, 2015

This is my ninth installment to a series on big data and how wide-ranging and even globally significant insight can be developed out of small-scale local and even micro-local data (see Ubiquitous Computing and Communications – everywhere all the time 2, postings 265 and loosely following for Parts 1-8.)

I have written a number of times about Big Data in the course of assembling this blog up to here, and both in this series and in passing in a wide range of other postings and series too. And most recently, and certainly in this series I have focused on what can be considered the big picture side to this growing information capability. I wrote in Part 7 and Part 8 of this series, as an extreme for that approach, on the global have and have-not consequences of uneven access to this type of information gathering and utilization capability.

I turn in this posting to a more hands-on and even utilitarian set of issues that go into making big data per se workable at any level. And I begin with the absolute fundamentals:

• Big, in big data is both a defining strength and a limiting weakness.

The strength side of this is the one that is essentially always noted first in any discussion of this capability. If you have all of the data that you could need in order to descriptively and predictably model a system under consideration, you can more effectively and I add proactively respond to it and both strategically and operationally. This holds true whether you are trying to develop a more actively effective sales campaign for selling merchandise or a more effective political campaign for winning elections and bringing your candidates into office. It is true when you are gathering and utilizing epidemiological and related data with a goal of improving public health. It holds true for a truly open-ended range of contexts in which data can offer value and comprehensive data can offer whole new levels of such value.

Big is also a potential trap, and to illustrate that point with a set of working examples that I have been developing in some detail and for several years now, I cite the challenges to national security that are created by open-endedly and indiscriminately gathering in any and all possible surveillance data and from, and about everyone. (See for example, my series: Learnable Lessons from Manning, Snowden and Inevitable Others at Ubiquitous Computing and Communications – everywhere all the time 2, postings 227 and loosely following for Parts 1-19 as of this writing.)

When you gather in any and all possible data, you run the risk that you not be able to find let alone actionably organize and analyze the right data when you need it in order to gain benefit from it. And this holds true no matter how carefully you plan out and carry through on how you organize this flood of data into your databases, or how carefully and thoughtfully you connect your databases together or provide access to them and their informational resources.

• As soon as you have to allow for novel and disruptively unexpected data analysis needs, you face the potential for discovering after the fact, that your a priori data storage and access models were limiting and restrictive
• And that these limitations prevented your being able to make the right actionable decisions when they were needed – and even when you had all of the necessary data at hand so that in principle you could have made the right analyses and decisions, and you could have taken more effective action.

To take that out of the abstract and with a very well-known governmental example, US government agencies knew about the Al Qaida terrorists who hijacked those commercial airliners in September 11, 2001 in their attacks on the World Trade Center and the Pentagon and well before these attacks took place. It was known that they sought out multi-engine jet aircraft flight lessons from flight schools in the United States, and it was even known that at least one of these “students” told their instructor that they did not need to learn how to land a plane – only to fly it. The data necessary to preemptively identify this threat and forestall this attack was all there. But it was lost in a widely distributed network of disconnected databases that were closely guarded from outside access by the government agencies that variously owned it – and even from fellow government agencies. Big and disconnectedly distributed literally killed there. Now, at the risk of drifting too far into a digression, the US government has learned a lesson from that, that all but infinitely bigger Big Data is the automatic cure for preventing any recurrence of 9/11-like attacks on American interests, and certainly when all of this new data is flooded into what is at least formally one single huge data repository.

But my real topic here is Big, in big data and on finding those essential data points and details, and the patterns that they potentially represent and in a timely enough manner so that knowing them can make a positive difference. Big makes this finding possible. Big per se makes this usability that much less likely too and certainly when big data means Big for the sake of big and as an end-goal in and of itself.

I am going to continue this discussion in a next series installment with a look into how Bayesian analysis and approaches that stem from it can be used to more effectively manage and work with large-scale data. And in anticipation of that, and I add in keeping with my ongoing focus on innovation and the disruptively novel in this blog, I will discuss how the unexpected and emergent and the possibility of that impacts on any use of big data, and on capacity to make effective use of any data resources of any scale. Meanwhile, you can find this series and other related material at Ubiquitous Computing and Communications – everywhere all the time 2 and also in my first Ubiquitous Computing and Communications directory page. And I also include this series in my directory: Reexamining the Fundamentals.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: