Putting Big Data in Context

While futurist Ray Kurweizel and Moore’s Law gets all the headlines, over the years there has been a lot of interesting research and creative thought given to the idea of technological innovation and its implications on the need for human involvement in complex decision-making. In the era of Big Data, when social networks capture our conversations, likes and ideas like never before and sensor networks, or the Internet of Things, indexes more of the world around us, the fastest systems have access to more of the raw fuel, in zettabytes of new data, needed to make increasingly more complex decisions. But, does that mean smart systems will soon replace human decision-making?

In the mid 1980s, Mathematician and novelist Vernor Vinge introduced the idea of exponentially accelerating change, which is, at its core, a concept of rapid technological innovation separated by shorter time intervals between new advancements. Complementary to ideas like Moore’s Law and Kurweizel’s Law of Accelerating Returns, Vinge’s concept argues that eventually the rate of technology innovation will far outpace the evolution of human intelligence. While Vinge often paints an ominous view of such change, if we’re smart about managing the innovation, our unique human way of thinking can help put data in context, where human exploration is exceedingly enhanced when combined with machine insights. This type of collaborative relationship between man and machine has the potential to thrust us into a golden era of progress and productivity never before witnessed in human history.

But, there are some problems to watch for. One is the overreliance on what data, whether big or small, tells us. Interestingly, popular commentator and New York Times OpEd Columnist David Brooks, who typically focuses on global economies and politics, penned a great column that gets to the core of this issue. In the article titled “What Data Can’t Do,” Brooks proposes the limitations of Big Data. Some of which I agree with and others I don’t, but that’s for another time. His underlying point though, which he also explores in another recent column, is valid. He points to the fact that the best decisions possible are made with a combination of data analytics and human intuition.

I couldn’t agree more. There is a great debate today in machine learning circles about the nature of the human brain, and whether our minds can be reduced to a system of symbolic representations that can be ultimately replicated by machines, or whether, as contemporary philosopher Jon Searle argues in his famous Chinese Room argument, that software simply cannot replicate human imagination, intuition and consciousness. Whatever the case, it is clear that over-reliance on data can be misleading.