Make it a reality show and you’re good.
1 Like
Well, speaking from within the social sciences, I’d say that most social science is presumed to be universal by people who don’t know much about social sciences. In my field when I was in graduate school there was intense conflict about every damn little thing - to the point of insanity. And cultural context was one of the many nuances that make it damn near impossible to draw conclusions about anything when talking about humans.
That doesn’t mean there isn’t scientific value in building our understanding of how people interact and how we operate socially. It just means it’s really hard to do. In comparison to physics or even biology we are just now sorting out the periodic tables and discovering leukocytes, and there are still a lot of people searching for phlogiston or advocating a good healthy bleeding to set the patients straight.
The problem is when so many people are making vast policy and political prescriptions based on nascent sciences. Economics as a discipline is like a two year old - full of potential, slowly discovering its own limits and the world, cute at times, with a shocking capacity for violence and unable to control its bodily functions. No way does the toddler get to drive the car - and yet we make huge, life and society altering decisions based on their latest or most prevalent ‘discoveries’.
5 Likes
Actually, the Milgram experiment is one of the most replicated experiments in psychology. the conclusion was so unexpected that the various parameters that lead people to obey have been extensively studied. We now know that you get lower obedience if the experiment is run by a less prestigious organisation, if the “scientist” is not wearing a labcoat, or if another “participant” in the experiment quits fist.
Hopefully, we also know that if people are given an opportunity to cheat- they avoid delivering the shocks and act as if they have done so.
Less hopefully, there was one variant of the experiment entitled “Obedience with a genuine victim”, where a puppy was the recipient of (non-deadly) electric shocks. Most people still reached the end of the experiment.
Of course, it would be impossible to get these experiments approved today, but this is an example of really good replication of data. It’s been done multiple times, with varied parameters, and strict controls. If only everything else were tested as thoroughly.
5 Likes
People don’t often replicate actual experiments. When they do, it is not always the shortest path to the truth.
There was a neat essay which I thought was in “A Random Walk In Science” but I haven’t found it there, on how for several decades many decades people measured the speed of light but always came up with something that followed Michelson’s latest figure; until a completely new way of measuring came up with a conflicting figure. There was also another experiment where people replicated Millikan’s oil drop experiment, and someone complained that Millikan must have faked his results because his errors were too perfectly distributed (later disproved).
The real test with most of science is not whether it will fit with what everyone else is doing. Piltdown Man was not disproved from many years, but it because increasingly obvious that it was an outlier, if not a fake. I did my PhD trying to measure voltage structures on superconductors that most of the other people in the world though should be spirals, and produced evidence to show they were spirals. My PhD had some evidence that they were much more complicated things that depended on a few seconds of arc alignment between the material surface and the magnetic field. It was hard to write this up because I had not got a positive result to report, and it is much easier to write up something when you have a good story with a good ending. I doubt if my PhD had much effect, but spiral fields seem to have gone anyway.
Social sciences experiments do not overlap with each other the way most other science does. If there is little overlap, and there seems to be a strong reaction against repeating experiments either, then this isn’t healthy.
1 Like
I’m curious whether your field is psychology. I mean Anthropologists don’t have any trouble seeing how huge the effects of culture are. Sociologists are somewhere between anthro and psych in this regard, but closer to anthro. It’s psychologists who seem to have a really hard time grasping that, in my experience.
I’m pretty much with you on economics. More than any of the other social sciences, it affects the very system it’s trying to study with its own ideology. Also, large proportions of respected economists have publicly made predictions about their own field that were completely fucking wrong multiple times in my adult life. An episode like that would shake a real science to its core and shatter its paradigm. In these cases, however, those exact same people just keep repeating the exact same bullshit like nothing ever happened. This makes it clear that, for most of them, their views on economics are pure ideology and have nothing to do with science at all. (I’m not saying that there aren’t economists out there who do good work. You can often identify them by the disclaimers they put in front of it specifically eschewing any claim to absolute, objective truth.)
4 Likes
As much as I like Maggie’s science reporting here on BB, I find her headline for this article a bit misleading. While the fact itself (not nearly enough social science findings are replicated) is true and lamentable (and has been lamented numerous times), the linked article actually reports a laudable effort to do just that. This should have at least been mentioned in the blurb.
And, as other commenters have already pointed out, this is not limited to social sciences and is not caused primarily by how the researchers “do” science but has a lot to do with publication and grant policies. As it stands, I read the title and blurb to carry the all to prevalent “look, social scientist just don’t do real science” undertone.
Disclaimer: I am a research psychologist.
3 Likes
On my “if I had a time machine” list, one item is "go back and find the first Humanities professor who publicly spoke the phrase social sciences for their discipline and kidnap them. I’m sure we could survive the resulting paradoxes just fine.
As for economics, at least economists have the decency to refer to their practice as “the dismal science”.
1 Like
heck, engineering has a lot of implicit replication of work, to the point that it probably has more model-checking in practice than the sciences do. if component A doesn’t actually work, and you use it as part of project X, and project X fails, there’ll be hell to pay.
i’m totally baffled as to how acoustics can possibly have no replication of work.
1 Like
to be slightly pedantic, repeating a trial at the same lab only reduces measurement error; it doesn’t address experimenter bias, faulty assumptions, faulty equipment, etc.
There’s gonna be job openings for professional curmudgeons? Sweet!
2 Likes
The BBC did just that with the Stanford experiment 10-15 years ago as I recall. It was some of the better kind of ‘reality’ TV.
It often seems it would be a great deal nicer nicer if a significant portion of them would just pack it completely the fuck in.
People do similar experiments, but they rarely do the same experiment with the intent to verify another experiment. For example, someone came out with a novel new array type, called a coprime array, and there are multiple researchers looking into them right now, but there’s relatively little overlap in their research. One researcher that I know of is using them to measure arrival directions in air in a concert hall. Another researcher is evaluating beam patterns for discrete sources in deep water. Another researcher is evaluating beam patterns for discrete sources in shallow water (which is actually a different problem).
To me, that’s not replication. The results of each of those lines of research will be distinct, and you can’t necessarily invalidate one using another.
1 Like
eh, it’s close enough for me. eventually a theory will probably unify these disparate trials, and that can be tested if necessary.
this doesn’t seem like ‘experimentation’ in the formal sense of hypothesis testing. it’s more like people are experimenting to find the properties of a novel invention; there isn’t anything to replicate yet.
there’s nothing wrong with the latter, of course; in fact it’s probably more important, and usually more fun too. i’m just saying that i think it’s a different domain than what social scientists do.
I hope one of these spurious conclusions is the one about people who get confronted with evidence that what they believe is wrong and thus believe their original wrong belief even stronger. Because that makes me want to give up on humanity entirely.
Neither does taking one measurement and calling it a day.
The article is about the social sciences and the reproducability of hypothesis-driven scientific research. Your emphasizing reproducible measurements without a hypothesis, which is what I think of as technology rather than science. There are people doing science with little more than a steno pad and #2 pencil, and there are large labs where there is lots of technology but no hypothesis-driven science at all.
Probably most important discoveries start as an outlier. We plug along with phlogiston and ether until something is discovered that those theories can’t explain. In five years, string theory may have gone away because it can’t be reconciled with some discovery in a different field. But the new outlier needs to be supported by its own theoretical basis, and that creates the falsifiable hypothesis for testing. Sometime the outlier goes full circle from being the Discovery That Changes Everything back to practically nothing, like cold fusion or polywater.
i’m not emphasizing any such thing.
what’s your point anyway? all i said was that using statistical controls for measurement error doesn’t correct for systematic bias. this is a universal point, and has nothing directly to do with whether something is “really” science or not. i don’t know where you got this “one measurement” thing from, but it’s weird.
anyway, just because you’re doing formal hypothesis testing doesn’t make something science. there’s plenty of bullshit which can be formalized as hypotheses, and as a statistician i see plenty of it. conversely, there is a lot of extremely interesting work in many fields which is not (yet) formalized hypothesis testing.
1 Like
Most results in ‘hard’ science are not replicated directly, but they underpin the next experiment. If a result was false in ‘hard’ science it would be exposed not through direct failure to replicate, first of all, but because the subsequent experiments that assumed the truth would fail.
How many social science results underpin further experiments or studies?
3 Likes
My dream job is “Wise Old Drunk.”
3 Likes
