Platt Perspective on Business and Technology

Big data and the assembly of global insight out of small scale, local and micro-local data 3: demographics-based and other descriptive and predictive models

Posted in business and convergent technologies, reexamining the fundamentals by Timothy Platt on December 29, 2013

This is my third installment to a series on big data and how wide-ranging and even globally significant insight can be developed out of small-scale local and even micro-local data (see Part 1 and Part 2.)

I focused in Part 2 on the rapidly evolving fine-grained focus in what local and micro-local mean and with the working example of:

• Progressively focusing in from a larger local community to smaller neighborhood communities and then to the individual level,
• And even to a specific-context level where single individuals can be treated as if containing within themselves distinctly separate micro-locals depending on where they are and what they are doing.

I stated at the end of that series installment that I would turn here to consider how this flood of data is used, and both at a local and micro-local level and for purposes of more global analysis and insight. I will do that, but preliminary to that I would more fully discuss our current state of micro-local and what that functionally comprises.

Basically, any data is potential fuel for Big Data. You go into a restaurant and while there take a photo of your meal, and a photo of your table and the dining area that you sat in while eating there. You post them on a crowdsourced review site such as Yelp along with a review, and on your Facebook page. Some of your friends and online acquaintances comment back, one offering equally localized and specific information about another restaurant in the same neighborhood and what they have eaten there. And the overall and even globally reaching conversation continues, this thread of exchanges matched by a seemingly endless number of other, equally focused, with their micro flows of photos and text and of very context-specific data. Micro-local is about the individual person, but it is also about the individual business and with specific experiences and interactions with both and with all of that enriched by progressively finer and finer levels of real-time data. And micro-local is about the layered and interconnected accumulation of all of this fine-detail data.

• In its extreme and as a perhaps ideal, big data and its accumulation of raw data seek to become in effect, a digital representation of all of us and of all of the world around us, as we interact with it and are affected and influenced by it.

And with that perhaps extreme representation of big data in mind, and as an intended goal for big data aggregators and users if nothing else, I come to the issue of using all of this, and of making big data a tool rather than just an accumulation of so much clutter.

Big data seeks to accumulate virtually everything, or at least seemingly everything. The value in it is not in its open-ended accumulation, but rather in how that mass of raw data is used and converted into actionable knowledge through that use. And that process can be divided into two components that are closely interconnected with each other:

• The formulation of hypotheses that would be tested against selected subsets of this accumulated data, treating big data as a source of empirical evidence for use in what amounts to scientific experiments, and
• The SQL and related search queries that these hypotheses are functionally translated into when converted into a form that can be used to select out pertinent data for hypothesis validation purposes.

Big data content is filtered out and organized according to hypothesis-based conceptual frameworks, to build abstract models, and at levels of resolution that range from predicting the preferences and behavior of individuals, through to modeling the activities of entire large-scale demographic groups.

Wide-ranging and diversely sourced data is used in all cases. When you go to a site like and conduct a search on their web site for some specific product or type of product, you are generally offered suggestions listing products that others have considered who have also looked at the same or similar products to what you have just searched for. And Amazon’s determination of what to show you from other customers’ searches is shaped by both your own searches and purchases on their site, and on activities of others who share what are predicted to be pertinent details in common with you, and who as such are predicted to belong to a same marketing demographic group as you. Your behavior here and whether you look at, or look at and purchase from these suggestions, or ignore them is then used to both further refine their data profile on you, and to further refine their models of the demographic groups that they associate you with.

• So along with its raw data, big data integrally contains interconnecting metadata that organizes its raw data content according to demographics and other models based on findings already developed from it. And further use of this overall resource simply expands the amount and complexity of the organizing metadata already in place as well as adding in new raw data, with both increases serve to expand a big data repository’s overall usability as a comprehensive digital model.

I am going to continue this discussion in a next installment where I will look into where this is all headed. In anticipation of that, I will be discussing forces that tend to separate data and partition apart what can be accessed and coordinately used, and forces that tend to bring all of this data together. Meanwhile, you can find this series and other related material at Ubiquitous Computing and Communications – everywhere all the time 2 and also in my first Ubiquitous Computing and Communications directory page. And I also include this series in my directory: Reexamining the Fundamentals.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: