What is Big Data and when will it be Smart Data?

Big data is cell phone users having an average of 100 interactions with their phone per day, all of which generate computerized records (100s of trillions of records). Big data is every financial market transaction, every passenger on every airplane flight, every shipped container, every transportation conveyance, every tweet, and every Internet post (all in the 100s of billions or trillions of records). Every transaction for all time.

One area of long-standing data interest is mortgage statistics since mis-estimating prepayments can cost investors billions of dollars. This raises the question of how prepayment risk is still being mis-estimated. Irrespective, mortgage data is one of the fastest growing kinds of data, both by row and column of tracked data, growing at more than 2x Moore’s law on a log chart (Moore’s law reflects the hardware on which the data is stored and manipulated (algorithms somewhat fill the gap)). This begs the question of smart data rather than big data.

There is much talk about all types of data growing (and data scientists being the biggest category of job growth), but the size of big data should surely be one of its most basic attributes. What is much more relevant is the value that big data provides through its use. For example, how has having more rows and columns in mortgage-tracking spreadsheets improved (if at all) prepayment prediction?

Like genomics, many big data problems are in the early stages of ‘the diffs,’ not knowing which part of the data is salient to keep out of the 99% that may be useless. ‘The diffs’ are the differences, the differences between a sample data set and the reference/normal data set that constitute salience and allow the rest of the data to be discarded.

Subjective Experience and the Existence of Free Will in Bergson

With burgeoning progression in neuroscience projects across a variety of fields including stem cell generation, brain scanning, and natural language processing, the free will / determinism debate remains vibrant. One resource for understanding the problem is French philosopher Henri Bergson and his claim that free will exists, and can be understood through how time and free will are connected.

Henri Bergson lived 1859-1941. 1900s. He was well-known in philosophy and intellectual culture more broadly in the early 1900s, including for anticipating quantum mechanics 30 years ahead of its discovery due to his assessment of time as being asymmetrical. In the 1960s, the French philosopher Gilles Deleuze reawakened interest in Bergson, highlighting the importance of Bergson’s concepts regarding multiplicity and difference. Now Bergson continues to be relevant to neuroscience and other areas interested in the understanding of subjective experience, free will, and mind/body dualism. Bergson published three masterworks:

• Time and Free Will: An Essay on the Immediate Data of Consciousness (1889) arguing in favor of free will 
• Matter and Memory (1896) resolving mind/body dualism with a larger problem frame taking both dimensions into account 
• Creative Evolution (1907) linking the idea of the time as energy and the energy of time to evolution 

Linking Time and Free Will 

According to Bergson in Time and Free Will, and as explicated by Suzanne Guerlac in Thinking in Time, we cannot treat the inner world of consciousness and subjective experience with the same model we use to understand the physical world. We need to purify concepts from their objective scientific use for the purpose of examining subjective experience, where the important features are the intensity of qualities, the multiplicity of overlapping mental states, and duration, the lived experience of time. Time is a force because it has a causal role in experiences not being the same each time, or over time, and in allowing experiences to accumulate through memory. Time is therefore a force, but an internal force not subject to the laws of nature as external forces. Exactly because time is not governed by mechanistic external forces, it allows room for the exercise of free will. The force of time makes free will possible and we exercise it when we are living in time, tuned into our subjective experience, and acting passionately and decisively. A more accurate conceptualization of our freedom is not in deciding between two alternatives but rather in experiencing free actions carving themselves out of our hesitation as we plunder though the constant becoming of life. 

Further explanation: YouTube video

Technology vs. Free Will? Tuning into the Internal Qualitative Experience of Time

Henri Bergson, the French philosopher, was living and writing in a time (early 1900s) similar to today where the furious pace of innovation in science and technology was promising to elucidate the deepest secrets of the world such as how the mind works.

Determinism Victory from the Application of Quantitative Methods? 
As the social sciences gelled into departments of academic study unto themselves and sought to apply techniques from the hard sciences, Bergson became concerned about the potential loss of free will and a victory for determinism. Humans might be reduced to billiard balls in the sense that if human behavior could be predicted mechanistically like that of a billiard ball, it would mean that humans would lose their liberty and free will.

Doublings: Experiences with both Inner Qualitative Subjectivity and External Quantitative Objectivity
Bergson proposed that there are several concepts that are different in our internal experience (qualitative, subjective) than in our external experience of the world (quantitative, objective). This difference between internal and external experience (called a doubling) exists in areas like time, intensity, multiplicity, duration, self, and consciousness. The external aspects can be measured quantitatively, but the internal aspects cannot, they are states that overlap, merge into one another, and emerge and recede dynamically.

Prescription: Tune into the Qualitative Aspects of Inner Experience
To Bergson, freedom is most visible in spontaneity, the ability of a person to choose spontaneous action. To maintain free will, one should tune into the qualitative aspects of internal experience, understanding concepts like time as a qualitative overlap and ebb and flow of states dynamically, energetically. Bergson’s nomenclature for inner qualitative time is ‘duration’ as opposed to external quantitative ‘time’ – this is the difference between the sense of waiting for a train to arrive (qualitative) versus the time elapsing on the clock. Being attuned to the qualitative aspects of time, one can live more spontaneously.

How to be a Philosopher

One of the great values of philosophy is that it can be a helpful tool for thinking through any question of deeper puzzlement. Philosophy is needed now more than ever in a world that seems to be changing quickly, and the field could enjoy a renaissance in wider popular application. The practice of philosophy need not be confined to experts but is accessible to all. Here is a short-list of how anyone can be a philosopher:

Level 1 

• Study the ancient texts 
• Claim a new read on the old texts 
• Claim that no one understands what [old philosopher #1] really said/meant and explain it yourself

Level 2 

• Find basic concepts in philosophy that no one has really thought through yet such as performativity ("...really performativitie...") Examples: Heidegger’s being and metaphysics, Deleuze’s difference 
• Invent strange neologisms (really neo-spellisms) by spelling existing words in new ways that no one can understand, and use them brazenly without definition: “…this is to say not just Possibility Space, but really Possibilitie Space…” or "the conceptologie here..." Examples: Heidegger's existens and phenomenologie, Deleuze's differance and differenciation, Derrida's linguage
• Invent new compound word combinations: "...the singularity of the self-other and the plurality of the group-crowd..." Examples: Foucault’s power-knowledge, Heidegger’s standing-reserve 

Level 3

• Submit a paper to a philosophy conference with a title like: "Rethinking [ancient philospher]'s [important concept] in [modern philospher #1] and [modern philosopher #2]"
• Offer a critique of the history of Western philosophy and propose a new/real metaphysics (examples: Heidegger, Deleuze)