On Big Data's shrinking returns

[Read the post]

The reason that our relatively small brains can cope with our environment is that they are very good at deciding what not to notice (even our dog does this - on successive days of Glastonbury our neighbours returned around 2a.m.; the first time he made sure we were awake and paying attention, the fourth time he just growled softly for a second). But for this vision couldn’t work because the bandwidth of the eye to the brain is kilobits per second, not megabytes.
So advertising eventually just fades out and we get good at detecting the content in the background noise. Targeted ads are no use if you don’t notice them. This is why there are periodic patent proposals to find a way to force people to look at ads, forgetting that this will be totally counter-productive.
At some point the web is going to have to move to a paid for system. But if it does it will shrink. How many people would actually pay for Facebook? - whereas they will pay for Linkedin, because it delivers something valuable.
Facebook has recognised this and is trying to set itself up as a cut down Internet supplier in less developed countries. Good luck with that; the local competition don’t buy entire rows of houses to keep their personal privacy, they will always be able to undercut Facebook on cost.
Google services? Yes, I’d pay for them because they deliver something actually useful. (The British government’s Cabinet Office agrees with this assessment.)
Perhaps we will end up with a world divided between Microsoft, Apple and Google, each of which has a revenue stream that isn’t just dependent on selling data to advertisers, even if Google’s is at a fairly low base at the moment.


Remember when Xynga’s “social games” like Farmville seemed to colonise the limbic systems

I think you mean Zynga

1 Like

Advertising doesn’t really work on a ‘click and buy’ level. This is the dirty secret they all fool themselves into trying to ignore - Ads do two things:

  • Get your product name out there and let people know that it exists - sometimes this results in a click to learn more, which isn’t a buy

  • annoy people

The only time I look at advertising is when I’m actually shopping for a product in question - that is say I need a new widget - I’ll search for that widget, and pay attention to adds/articles/etc. that talk about widgets and who makes them, and figure out which ones I want.

You know what I find invaluable? User ratings, and then only when they are verified purchasers - because the number of robo-reviews is getting overwhelming.


That’s a funny fucking joke! GOOD ONE!


While daydreaming about surveillance, privacy, and ways to protect myself without unplugging, I wondered how effective it would be to throw up tons of chaff. Basically - profiles that I didn’t use filled with erroneous data, random likes and follows, etc.


I have to wonder what they think of me. I don’t have a Facebook account, and I use Startpage instead of Google - it forgets everything. My son googles on my laptop once in a blue moon, sometimes for the oddest things. I’m pretty sure we’re on the list of pirate movie watchers now.

There are many services that most of us take for granted that do depend on revenues from advertisement. I understand that Google may actually find money from elsewhere, as it’s branching out into being an ISP and even mobile payment operator. Facebook on the other hand may suffer immensely. Anyhow if advertising no longer pays the bills, what will the internet become?

I’m not entirely sure how to take this article. It seems to conflate the incredible difficulty of building ad recommender systems for advertising where there is no explicit intent like on Facebook to the failure of Big Data as a whole. That’s just complete nonsense. Hell, it seems to confuse Big Data with archives of personal information when that is simply one kind of dataset.

And Big Data has failed to predict human behavior over long terms hasn’t borne scrutiny? I have no idea why you would think that. We’re actually pretty good at predicting behavior of populations and its used for everything from supermarkets knowing how many avocados to stock to pricing of airline tickets to fraud detection. We mostly just suck at individualized predictions (our most accurate algorithms simply do not scale which is why even book recommendations are awful since the computing power to calculate good ones on a large scale would be truly ridiculous).

And no, the CPM rate for Facebook ads even if they had some magic algorithm wouldn’t necessarily climb into the stratosphere. In order for that to happen, they would have still need to have the right ads to actually show the user (which requires a ridiculously large supply for every conceivable situation - especially if your magic algorithm works so well you never show a person the same ad twice) and advertisers willing to pay the rate (which is limited to the lifetime value of the prospective customer).

Seriously. I hate articles like these. Big Data is not a boogie man. Most of the time it isn’t even used for processing personal data and even when it is, it quickly becomes impossible for any human to understand and is only useful when compared to other people’s data.

That’s not to say people shouldn’t fight for privacy, but stop treating the phrase “Big Data” which simply means “too much data for a single computer to process” to mean personal data.


I thought about this for a moment…and then I thought about my cable bill.

Sure ads pay for things on TV as well, but the large chunk of change I pay TWC every month also pays for a good deal of it. That other large chunk of change I pay TWC for my internet, you know in theory, could actually go and pay for the internet…

And in reality if it comes down to a pay to use type of internet I’m probably not paying. I think you’d start seeing a lot more small BBS style sites pop up, but no I’m not paying to use Google or Facebook.

Really? My nephew runs a small business and he says Linkedin has enabled him to make contacts and get business which he could otherwise not have reached, I know people who have found jobs through Linkedin, and people have used it to ask me for references.
YMMV, but there are surely people out there who are willing to pay for Linkedin, which is why (unlike the social media websites) it makes money.

Micropayment schemes haven’t worked out well so far.

Woah, way to misunderstand regression to the mean, Cory. That’s not how regression to the mean works at all. The point of regression to the mean is not that factors like repetition and competition would eat away at the usefulness of techniques. The point is more that phenomenons come to our attention because they are an extreme value, and independently of any underlying cause or long term pattern, the chances are, next time we look at it, the value would be less extreme. Even if the extreme value actually turns out to be significant and important!

We must therefore be very wary of drawing conclusions from such declines, because the fact that we see them is due to the way we pay attention to such processes in the first place.

I think big data has great promise and it has only barely been tapped in a few select instances–virtually none of which have been directly applied to advertising. While it’s not wizardry to predict people searching for “used cars” are looking for used cars, it is impressive and maybe valuable information to a company that has couples as their demographic to know when two people are about to start dating before they know it themselves. Say right then you start seeing more ads for Ben and Jerry’s, Komforte Chocolate, and the sort of restaurants that both you and the person you’re likely dating enjoy and you never have to even announce your relationship for facebook to know with sufficient certainty. Likewise it might be advantageous to know when two people are about to break up without them explicitly announcing it. Suppose then the ads for Ben and Jerry’s are replaced by ads for half gallon Mayfield’s icecream, Hershey’s chocolate, cheap frozen pizza, and vain dating apps like Tinder. :stuck_out_tongue_winking_eye:

The biggest and juiciest data is still untapped and will remain that way until machine learning becomes better. When people can be auto identified in photos when they’re not even facing the camera and more importantly, when machines can predict which products you own based on what’s in your photos with you above the norm, they can offer truly intelligent marketing. Creepy? Yes. Will it probably still happen because economics always wins in the long run? You betcha. The fact that today we have AI that is beginning to, in plain English, identify complex objects that take many different shapes (like cats) and describe their relationship with other elements in the photo, tells me that within 7 years we will be using this tech to make money. Big data is still valuable but we’re still mostly processing it with mid 2000s hard coded algorithms that have recently written instantiations.

Things will change with machine learning.

1 Like

I have been frequently startled by targeted advertising. How do they know that I already bought that? Eerie.


You make a good point, except about the definition of “Big Data” to most people. Big Data is the big corporations that collect the data, NOT the data itself.

Well that’s just ridiculous. Under that definition, most large companies would be Big Data from banks to aerospace. Cory though seemed to be using it to mean the data itself since he talked about the hype cycle - well that and the ecosystem around it to use it.

Enh web browsing history, what not. I am always surprised to find ads for weeks for something I already purchased at amazon or such. Which is dumb cause oh, really already got that why do you think I need more?

1 Like

LinkedIn is fine as a virtual Rolex/Yellow Pages. The problem is that it tries to bully all of its users into “increased engagement” all the time with its articles and “update this!” reminders. It’s become yet another site where a lot of people feel obliged to have some sort of presence, but few people actually enjoy it.

1 Like

Fair enough. I never enoyed interacting with agencies, but you have to do it sometimes if you want to be gainfully employed.