Big data's diminishing returns

Have you ever read novels from over 100 years ago? Notice how often people were afraid something embarrassing would become public? People would commit murder to hide the fact that they were homosexual, had an illegitimate child, or any number of little secrets from their past. The only way these novels have any tension about people's secrets is if we can put ourselves in a 19th century mindset.

Okay, so there are still plenty of people alive today with a 19th century mindset. Every year there are a lot fewer of them. On the flip side, every year we have young people who are growing up with no sense of privacy. Everything they do is on social media -including all the dumb things they do. Maybe especially the dumb things they do. They joke about doing things a Victorian would have paid blackmail money to keep hidden.

So what happens when these young people become old enough and interested in public office? Embarrassing things from their past? No big deal. Everyone has embarrassing things from their past. It may actually be suspect if someone does not have skeletons in their closet. It won't be normal.

There are two groups interesting in all that data: governments and business. Governments, by their nature, don't trust the public and want to get the goods on them. Business wants to know all about someone so they can sell to them -or to do things like deny credit to people with bad medical histories. That's just creepy.

So what has the government found out? They've discovered that a lot of people hate them, so the government spooks are feeling pretty unloved. They also want to discover dark things about folks. That only works as long as people care to keep those things hidden. Will the younger generations even care? Blackmail doesn't work when no one gives a darn.

Right now businesses are very excited by the promise of big data. Every tiny bit of information about someone has market value. Past performance is a good indicator of future actions. That works as long as everyone is safely in their ruts. Big data is only predictive for fairly short periods of time in a relatively stable society. When things upset the status quo, all old data goes out the window.

Now there are big data geeks who think they know how people will behave in stressful life changing situations. They may even have a small track record. Fine, but wait until truly disruptive things knock people out of their routine -repetitively. It could be anything from war, plague, a comet strike to a new religion suddenly catching fire. Mix and match for even more confusion.

Then you have the people on the web with a number of personas totally divorced from their real selves. No one on the 'net knows for sure if you are even male or female. Then there are folks who think it's fun to game the system by feeding erroneous data into the mix. In a world where no data is deleted, the bad data pollutes the stream forever. Go ahead, Google for things that you have no interest in at all. It's fun, like throwing a tiny monkey wrench into the system.

-Sixbears