Nice. âIn the national interestâ - it falls under that definition, Iâm sure.
Too much data? We must gather yet more data so we can do a meta-analysis to find out which data we should be paying the most attention to.
Better move from Cheltenham to somewhere with a bigger river, all those data warehouses are going to need an lot of cooling.
Orwell must be turning in his grave.
it is obvious on its face that if you're looking for a needle in a haystack, you do yourself no favours by making the haystack as large as possible.I don't think that's a good way of describing what mass surveillance does or how it fails. Obviously no matter your data collection methods (mass or otherwise), you're always starting with far more data than you need - but that's not a bad thing. That's why you depend on many different kinds of filters to eliminate unrelated data from your analysis.
Now, you could argue that restricting your initial data collection is a kind of filter, but itâs a pretty crude one. Thereâs no reason to think that your restricted initial collection has necessarily captured all available relevant data and excluded only irrelevant stuff. If you wanted to find out, you could also collect the stuff outside your focus, and then just design more sophisticated analysis tools to filter out the noise introduced by more data. I think this is what MI5 is actually saying: we have the collection down pat, but our ability to make sense of it is lagging behind. We need more funding for advancing our data mining systems and analyst budget, so that the nightmare of total government insight into all of society can be finally achieved.
We should not argue against mass surveillance on the basis that it doesnât work. It doesnât currently work, but it can, and thatâs what makes it so dangerous.
Sounds to me like an argument for an increased personnel budget.
Letâs keep making that haystack bigger! Weâre sure to find more needles that way!
But why do they need all available relevant data? If youâre trying to catch suicide gunmen you donât need every detail of their planning, you just need one relevant clue to cue further investigation.
This is a fairly common thing you learn in science. Most of the data you could collect is not very useful. Take samples, but donât just sample noise. Use your intuition to tell you what is useful to collect.
Of course these spooks arenât collecting this data to âstop terror attacks.â Theyâre collecting it for post hoc analysis. After they catch or identify an attacker using conventional techniques (warrants, shoe-leather) they will pick through their pile of data and they might find something that helps them finding conspirators. They could of course just get that data with a warrant, but theyâre lazy. They just want the computer to tell them who the terrorists are, rather than investigate. Their dreams of a perfect system greatly exceed their grasp, probably because the people who come up with these ideas greatly underestimate the amount of work and thought and experience it takes to make sense of the data, not to mention a failure to understand the fundamental logistical/statistical limitations of this approach.
Depending on what youâre doing it sometimes is easy to take a lot of data, more than you may need. So why not take everything? Lots of reasons, mostly logistical and statistical. Itâs easy to overburden yourself with too much data that you canât draw good conclusions from, or which you might be able too, but the effort involved would consume time and doesnât really advance the goals of the project. Use your intuition to collect the data you think is relevant, then analyze it the best way (which is often, but not always, the most simple way), and then draw your conclusions and move on to the next experiment.
Say youâre doing a time-course experiment on a machine in an automated fashion. The conversation might go like this:
Me: We tell the machine to take a reading every 5 minutes for an hour, which gives us 12 data points.
You: Well what if we tell it to take a reading ever 5 seconds, then we have 720 data points! More data!
Me: Yeah, but what will that accomplish? The reaction proceeds slowly, so weâre not going to see changes on a 5 second scale, so why bother taking that extra data?
You: True, but it doesnât cost anything to get that data, so why not?
Me: Because it does cost something, itâs more to analyze.
You: But the analysis is just a computer script right? So it doesnât take any more human time.
Me: True, but I still have to think about it, and thinking about 12 data points is much easier than thinking about 720. The real issue though is that most of those 720 points are not informative. Meaning, the differences between a reading at Time = 5:00 minutes and Time = 5:05 minutes will most likely just be noise in the machine, technical error, not due to the reaction itself.
You: But wonât the statistics âsort all that outâ?
Me: They could, but since the proportion of informative data points is so low, Iâm going to lose a lot of statistical power. You can get around this, but itâs going to need a potentially more complex model. Imagine: The difference between Time = 0 minutes and Time = 5 minutes should be pretty large, and we will see a change, but the difference between Time = 0:00 and Time = 0:05 will be small, or negative, even though the overall trend is positive. So comparing each point to the one previous to it will likely show no significant change, even though the overall trend is positive. Instead of just comparing each measurement to the previous one in a repeated-measures-based approach, Iâm going to have to fit a full regression on those points, which yes, happens on a computer, but itâs still more work. I may also have to fit a non-linear regression to get the same result, which is still more difficult and makes it hard to interpret.
You: Well then why donât we just measure every 5 seconds and then you can just pull data from every 5 minute interval, so itâs the same analysis.
Me: Thatâs no different than just doing what we were doing!
You: But then weâll have it!
Me: Yes, but if we collect data we incur an obligation to analyze it. Imagine defending this to a peer-reviewer. Are you gonna say âWe collected all this data, and then ignored 99% of it, the points we kept were chosen fairly arbitrarily based on what was easier to analyze.â Now we look like weâre throwing out potentially relevant data.
You: Well shouldnât we be capturing as much data as possible?
Me: No. We should be collecting as much relevant data as possible. Itâs a waste of time and money to collect useless data. We know this protocol works. It makes useful predictions, it tells us when this reaction is working or not and the rate at which it happens. This data successfully informs future experiments. Weâre doing an elegant experiment.
If we were having problems, then maybe we would take more data, but thereâs a fine line between useful data-collection and OCD data-collection frenzy.
You: Well I still think we should do it.
Me: Well then you can analyze the experiment, weâll need the results tomorrow.
You: But I donât know how to do statistics, or use the analysis software!
Me: My point exactly.
Why do I get the feeling youâve had that exact conversation at least once (and probably many, many times)?
I would prefer âinductive logicâ to âintuitionâ.
To paraphrase Shannon, simply gathering more data actually reduces the signal to noise ratio. Thatâs also the challenge for CERN. But the information scientists at CERN have an idea of what they are looking for, and a clever triage system. It is that which makes the problem soluble.
The trick is not to collect more data than you can analyse, until you have that method of triage which enables the data to be reduced automatically until the sign to noise ratio allows the extraction of information. Or to put it another way, prove your idea works before betting the farm on it.
I generally agree, but I donât think the distinction between âinductive logicâ and âintuitionâ is very meaningful in this case. If youâre doing something for the first time, you have nothing to base the logic on, just your experience and intuition. You might say that your experience with or knowledge of similar experiments is the âinductive logicâ youâre using, but thatâs just intuition isnât it?
that method of triage which enables the data to be reduced automatically until the sign to noise ratio allows the extraction of information
That makes a lot of sense for CERN, but writing software to do that for experiments you do everyday would seem like a lot of work, and would require a lot of specialized training. Itâs making a mountain out of a mole-hill. Most researchers donât have the need for that, or the ability to program that. Moreover most peer-reviewers wouldnât be able to criticize such a system. In fact Iâm pretty sure what youâre suggesting is what statistics already does, though not always explicitly (except in the case of an informatics-based approach, like doing model selection/comparison).
CERNs difficulties are not generally applicable to most labs, where youâre can rarely do the amount of replication youâd like to. Some people tend to worry a lot about whether they are using the theoretically-most-mathematically-and-statistically-perfect method while theyâre limited for financial/logistical reasons to only having 10 samples in their experiment anyway. They waste time and money on âquarterbackingâ their methods in theory while not just doing some experiments. Any chemist will tell you that you can do all the theory you want, but when it comes down to it the reaction will either work or it wonât. The only way you figure it out is by doing it a lot. Most of the PhD projects Iâve seen synthetic chemists do boiled down to "I spent 2 years working on a reaction, turned out the solution was to mix it up in a smaller container, and add the reagent drop-wise while shaking vigorously, then it works great. The final years wre spent doing 20 variations on the reaction and quantifying carefully. No amount of reading or modeling chemistry is likely to tell you thatâs the secret, you just have to do it.
I remember an organic chem lab I took. We banged our head against the wall for 3 hours with a simple reaction. âMix A and B, heat to 70 degrees for 10 minutes, collect crystallized product by filtration.â Tried 5 times, wouldnât work. So at some point I said âFuck itâ and just boiled the solution and got tons of product.
âWhatâs the deal!â I exclaimed to my lab instructor (who was, some chemistry PhD student), âIt says 70! but that doesnât work, did I do it right?â âDid you get the right product when you boiled it?â, she asked disinterestedly. âYeah I got lots, melting curves are just right.â, I replied. âThen you did it right, welcome to chemistry. yawnâ
Ah. I am sorry; to paraphrase Rutherford, I was writing about science, I did not realise you were writing about stamp collecting.
[edit - fair enough. Myself, I would say that practical chemistry has a complexity problem that is hard to control. But I still think your examples are induction rather than intuition, as in âwell, the last time this kind of thing happened we did X and it worked, try a bit of X.â
As for mistakes in textbooksâŚI do suspect that as with cookbooks, the authors have often not actually tested their recipes and mistakes get propagated.
And no, I donât really agree with Rutherford.]
Oh I see, you saw the word âscienceâ and then wanted to make a joke.
Pass.
You appear to be responding to me rather than @nimelennar.
The XKCD seems to me one of the very much less inspired efforts - the remark about stamp collecting was indeed Rutherford not Feynman, and the argument in the strip is pretty stupid - biology has only been able to progress because of the work of physicists, chemists and engineers, and these have become much more effective due in part to the longer life expectancy and so on brought about by modern medicine, in a virtuous circle.
I have edited my original comment that you objected to. It wasnât my intention to cause offence.
I donât think that Munroe was trying to slight physicists at all. I think that he was saying that he was just trying to puncture the egos of those physicists who slight other fields (like biology).
Nor did I, I thought the argument was stupid - both sides put up straw men. Iâm a bit annoyed with bad arguments at the moment because of the sheer hideous awfulness of the UK referendum âdebateâ - which seems to consist of racists shouting at bureaucrats.
But my reason for posting was that @clayton_coffman seemed to have got annoyed with the wrong person.
This topic was automatically closed after 5 days. New replies are no longer allowed.
