Sure life has changed in the last twenty years a lot. But apart from the obvious technological change and all these gadgets around us – there are far more dramatic changes in our social, economical and political behavior.

Statistics and mathematical problem solving has been around for ages – but with today’s huge databases, networks and extremely cheap processing power suddenly “smart” computer advice is shaping our everyday life …

Buy! Buy! Buy!

Your personal shopping assistant is ready

The most obvious and understandable of these computer aided decision processes is Amazon’s great recommendation system.

Amazon very closely watches your purchases – as well as saved items on your wish list, searches, clicked results, viewed pages and rated items. This data flows into personal profile – which is already sucking up a lot of your “private shopping process”. But this recommendation process goes further and crosslink’s your behavior and selected products with other peoples “tracks” in the database – and computes it’s recommendations to you.

Almost all popular sites do this: last.fm, YouTube, StumbleUpon …

So far it sounds pretty innocent.

Can we be good friends?

Computer says you like each other

But social engines like Facebook – and more important dating sites like eHarmony or PerfectMatch expand that idea into social decisions.

They all have a set of questions, rules and statistical predictions to play the perfect digital matchmaker.

That is why all these bombard us with tons of questions and keep tracking our behavior while watching other people’s profiles. Tracking every move and saving every click we make to create even better recommendations.

Sure most people still make friends in reality, but the younger generation currently redefines “reality” itself but relying more and more on connected gadgets to keep their social connections alive – and making new acquaintances with the help of social software.

This is friendship 2.0.

Mad Genius – the beautiful Mind of Game Master John Nash

The Math behind it all

Most of these engines work with so called “regression analysis“. It’s actually pretty old stuff, going back to the late 17th century.

It had already found many applications in science – especially in psychology and later in marketing. It later swapped over into military and political planning as well.

Therefore it’s closely related to Game Theory, which tries to predict strategic behavior within groups of people, armies and political systems. It was for example used to play computerized war games during the cold war and create the philosophical basis of such nice ideas as mutual destruction.

But these early computer aided tool lacked several components that are absolutely common today: lots of data, lots of processing power, a global network – and most of all huge amounts of feedback data.

Welcome to our global feedback collection center …

Feed me!

Compared to old computer simulations (trying to figure out if some data model or theory actually works) today you can shorten the process by almost instant feedback.

You can “model” your system and fine tune it based on instant feedback from users, sensor and computers worldwide.

Take once again Amazon’s sales recommendation function is a perfect example: User can actively tell the system what recommendations they dislike or find useful. And any sales generated by the system is a positive feedback anyway.

The same is true for social models: millions of friendships (or passive rejections) are “executed” every day on gigantic sites like MySpace or Facebook.

So “reality” heavily “feedbacks” the computer model.

But also the opposite is true: the better the computer model, the more it influences the users and therefore it’s own feedback. The more people own a certain product the more ratings and feedback it gets, the more it becomes a “popular” choice.

There is a point where the computerized recommendations and self improving models create their own “reality”, because their output is of such high quality or interest to the user that they are simply satisfied or overwhelmed by all the “offers” and recommendations.

Those damned Hippies even ruined today’s Internet!

Hive Wisdom and Crowd Sourcing

So much about computers – but humans also generate huge amounts of “recommendations” and data we rely on.

There is also the other new social phenomenon beside wide used computer based recommendation on the nets that has also it’s roots in the 1960’s (like game theory).

The Hippies relished the idea / spirit of collaboration, communes and free exchanges. Community, free exchange, openness …

It’s no surprise that many early IT gurus were heavily influences but these ideas, drugs and new age philosophy. Slogans like “information wants to be free” combined with “power to the people” are the ideological parents of many modern websites and data systems.

Early Web Communities like bulletin boards were built in that spirit. The whole Open Source movement is based on it as well. The internet itself is built without borders and the free exchange of ideas and data packages.

In early digital communities consensus was often built in forum discussions or bitter flame wars. It was just a war of words, but nifty observers could see patterns of behavior emerge as well as a new form of digital collaboration.

Do you Wiki? Our new global library …

Let’s work together!

It took some time to build digital workspaces that allowed group data collection, editing and fine tuning.

The best and most successful example of this is of course Wikipedia.

It is so far the most impressive digital collective. And it’s speed is breathtaking. It has amassed over 10 million articles in several languages in just under eight years. this is the Gutenberg Revolution on speed.

Currently we collect data like mad – and everything gets rated, photographed, reviewed and tagged by the digital Volkssturm: restaurant reviews, book recommendations, etc.

Nothing is safe – and companies like Google make huge profits from the willingness of the internet users to collect data and share it willingly with the world.

It like a giant data ant hill were everything is catalogued and sniffed at over and over again.

Stop sniffing my ass – or I’ll post you mother’s naked photos on MySpace …

Together we are smarter and can be easier analyzed by machines

So on one side we have millions of humans working and communicating using computers – which makes it easier for computers to watch and analyze these humans.

For normal web surfers it has pretty cool consequences: websites are much more useable and good stuff is much easier to find.

It’s an information paradise that is like a well trained dog bringing you your slippers and newspaper right to your bed.

Scanned, tagged and released into the Markets …

The Power of Data in the hands of Business

The power of number is nothing new for business people, bureaucrats and politicians. The Babylonians and Romans already had huge bureaucratic systems which collected data – mostly to keep control over their empires and collect taxes.

And the numbers always were important to any business owners, may they be Jeff Bezos or Marcus Licinius Crassus.

Computers have totally changed global commerce: today you can track any aspect of production, sales, inventory, finance and shipment of your business. Data warehouses are quintessential for companies of any size.

You can “tag” physical objects with RFID chips and sensors can “pick” them up where ever they go.

The same is almost true for customers as well. Thanks to credit cards most shoppers can be easily tracked and “computerized”. But even such a personal identification via credit card is not necessary. Because what you put into your shopping basket on a regular basis in your own rhythm tells a lot about you as a consumer. Computers are masters in recognizing patterns (of behavior) and tracking them.

If you are using the Internet to shop – it’s of course much easier to track your behavior and analyze you.

Such data collections combined with HUGE recommendation and analysis help companies to plan demand and keep their supply chain well organized.

That is why such modern marvels as “just in time” and “built to order” production are possible.

It’s a totally new transparency for business processes and customers alike.

The whole thing is called data mining – and keeps many middle management guys, IT nerds and Excel Jockey happy and employed.

Life can be pretty weird, data even more …

Decisions, decisions, decisions …

Thanks to more and more data collected from business, scientists and governments more and more aspects of our lifes and societies can be compared and analyzed.

The book Freakonomics (here is the official website) lists many amazing and stunning correlations that haven’t been “seen” before – because of the lack of data, computer model and proper analysis.

Scientists have always used and loved math, statistics and data models to explore and prove ideas. The old idea of the world formula that explains everything as as much a part of this “Cult of Data” as the old mechanistical idea that the world is built like a giant clockwork (therefore God is often called the (blind) watchmaker).

Today very complex data collections and simulations help us to tackle problems like climate change, but also run our networked financial markets.

Once again – like for business owners – computers help scientists and governments to make decisions based on data, find hidden relations in the “system” and give us a better understanding of the world.

We are creating virtual models of every aspect of our world – which we try to make machine readable by collecting huge amounts of data. We are building virtual systems to understand the real ones.

So computer recommendations are not just for Amazon users, but also for governments, business owners and scientists.

Give me data or give me death …

More Diversity or more Mediocrity?

But back to us mere mortals …

The problems with such tight tracking and feedback system is that they tend to emphasize popularity.

Amazon will always show you the best sellers first, because the data model and feedback make it a safe choice.

Sure all better websites have functions to show us new stuff or recommend special interest choices (translated: the weird shit, hardly anybody else but you likes) to us. The systems try to foster diversity in the face of group behavior and group “buying” pressure. Why don’t you buy what everybody else is buying? It must be good, it got so many great reviews!

In contrast to us humans huge data systems like Amazon can handle millions of items, opinions and ratings at the same time. They can cope with diversity – and thanks for modern technology are much more flexibly than early computer models.

Diversity is not the problem for computers. It’s us humans. And we are the ones that bring mediocrity into the system. If you read opinions on Amazon or any other “community driven” system you’ll find the same opinions again and again. So the “wisdom of crowds” of often just a repetition of the same old blabla.

Instead of “sharpening” and “smartening” up of “user generated content” it’s all just more of the same. Instead of focus, we see a blurring of opinions and recommendations. It’s more like a trend, less of a sharp guiding line.

Quickly – let’s follow the trend …

Are we just data sheep following the critical mass?

Although these data systems and computer generated recommendations are basically reflecting our herd / flock behavior once again.

But aren’t we social animals? Aren’t we herd animals, forming groups and societies around topics, interests and other people?

But when is a group really a group – or better say a trendsetter to attract more data sheep? This effect is called the critical mass – this is also an important effect in web business. Unless you don’t have enough web users on your web site it doesn’t generate enough data, group pressure and diversity to run by itself. You need advertising and other clever tricks to attract more users to finally reach that critical mass.

But the same is true for people or product recommendations in social networks and shopping sites.

All these data systems mostly concentrate on popularity and “mass” and not quality. It’s easier for web sites like Amazon or Google to tell you if this is a popular book or link – and not if it’s a good one. Quality and true compatibility are much harder to measure than simply the mass of feedback or clicks.

And what about Quality? Even when other users tell us with the usual five star rating if they liked a product or thought it was good – this tells us very little about the actual and factual quality of it.

I am data and on the internet – therefore I am!

There is a voice in my Computer telling me what to do

We humans are social animals. We rely on other people’s opinions and advice to make decisions or to learn.

More and more of this input comes from “digital crowd wisdom” or “computer generated recommendations and ratings”.

This is unique in history. We had never so much “machine” and “data” in our lifes.

Good data is certainly better than guess work. And our advanced society is only possible, because we had scientists collect data and prove ideas with solid facts.

But are we now slowly making a transit from solid facts on one side and warm human interaction to fuzzy computer models?

We already check social networks and lookup people via Google to learn more about them. The following generations will rely even more on such data representations and recommendations about people as well as products.

Initiatives like Open Social from Google and social networks like Facebook and professional networks like LinkedIn will shift our “social experience” to a more “data  collection social experience” instead of human interaction and exploration (“Why ask this stranger anything – I can find all about him on the Net!”).

A digital reflection of yourself …

Final Words

Human social interaction was always based on the reflection of our personality and action by others. In the future data collections and recommendation models will analyze and reflect us. Not because machines will take over the world, but because they are much better in collecting data, tracking our every move in a networked society and rating us without prejudice.

But most important of all: because we humans love feeding anything and everything into computers – and we rely on them to run our societies,  social interaction, businesses, sciences, economies and governments.