Most social science results have never been replicated

[Permalink]

You’d think that psychologists, of all the people in the world, would know that they might be fooling themselves, so replication, replication, replication. What the heck are they afraid of?

Next on the chopping block: economics! Where are the paired experiments?

8 Likes

Yah, on the rare occasion I am in a discussion with someone who defends or promotes an idea because of “the economy” I will always ask, in all seriousness, “The economy? What is that, to you & yours?”

That will stump even people who do know what they are talking about. Defining how a macro gonna crush yo micro is remarkably difficult & usually pointless. Trying to defend “the economy” is more often than not just a tongue stained darkly in kool-aid.

3 Likes

Yeah, but social science isn’t really science, is it?

1 Like

Replicating the Milgram experiment or Stanford Prison Experiment would probably get your butt thrown in jail.

1 Like

The wikipedia page on Social Constructionism is worth a glance because this is where you end up when there is an absence of data. Edit: I actually agree with a lot of what’s being said there - I’m not trying to pick on Social Constructionism itself.

I think the weakness of any science can be seen in the use of excessive jargon to obscure the weakness of facts such as in psychology. Edit: This is actually a good line:

In particular, they are, in Searle’s terms, ontologically subjective but epistemologically objective.

And I always like Karl Poppers line:

The development of thought since Aristotle could, I think, be summed up by saying that every discipline as long as it used the Aristotelian method of definition has remained arrested in a state of empty verbiage and barren scholasticism and that the degree to which the various sciences have been able to make any progress depended on the degree to which they have been able to get rid of this essentialist method .

3 Likes

By the principle that, if it has science in the name then it isn’t science, then no.

1 Like

But you’d have ample opportunity to study the sociology of prison guards!

1 Like

Just pointing out that, in addition to this, most social science is assumed to be universal despite not being corroborated with a cross cultural study. I’d say, before you try to claim universality, you should at least have people from a variety of social classes, representatives from a couple of different non-european cultures, and representatives of some tribal culture. Psychologists are the worst for underestimating the degree to which culture shapes people.

It actually gets worse. Clinical psychologists might as well be witch doctors. The disorders in the DSM4 are generally assumed to be something innate to people, but they are in fact almost completely culturally constructed, and the spread of western-trained mental health professionals around the world is bringing those disorders to people who previously didn’t have them:

5 Likes

No one in science actually replicates experiments, in any field. Good experiments are costly. They’re costly in terms of design, in terms of setup, in terms of time of highly skilled people, etc. Replicating someone else’s experiment saves you a little bit of that design time, but even the best journal article won’t give all the details needed to properly replicate an experiment, and there’s still going to be a lot of time just coming up with how to replicate a given experiment.

And in the end, there’s very little to gain from the replication. If you get the same answer, then your result won’t be significant enough to be published. If you get a contradictory result, then you’re likely in for a prolonged battle with reviewers who are (naturally) skeptical of a paper that directly contradicts existing literature.

As a result of all of this, the value gained from replication is low, and so it’s nearly impossible to find funding for such experiments. The only people who would fund such work would likely be corporate entities with a vested interest in such a contradictory result (eg, oil companies funding alternative hypothesis climate change research). However, that runs the risk of tainting the research, and there are no such vested interests in social sciences, that I can think of.

In my mind, there’s actually a reasonable solution to this quandary: graduate students. As per our traditional academic system, earning a Master’s or PhD in the sciences involves conducting and writing on unique research in the field. However, because many scientific disciplines are fairly mature, the research in those fields tend to be esoteric to the point of being useless. Case in point, student theses have some of the lowest impact of any published works. In essence, the only valuable product that comes out of a student’s years of research work is the researcher them self. The grant money that gets funneled into that student is largely wasted.

We could, however, solve both problems in one fell swoop. Create a new path to a degree by allowing students to replicate an appropriate experiment (or experiments) as determined by their thesis committee. The students gain firsthand knowledge of how experiments in their field are designed and carried out, and they make a meaningful contribution to their field by validating works that could be fundamental.

9 Likes

In the lab sciences, you don’t publish the replication, but there are plenty of times you have to replicate some older work first as the basis for your experiment.

3 Likes

It’s clinical trials that usually don’t get repeated.

In lab science, you are probably going to repeat an experiment five or more times you can at least add the standard error of the mean (SEM) to your graph.

Not if it’s part of business plan

What do you define as “Lab Science?”

My field, acoustics, an offshoot of Physics, almost never has replication of work.

1 Like

I was specifically thinking of materials science. Are you really saying that you don’t use other people’s work as the jumping-off point for your own work in acoustics, in such a way that if their results weren’t correct, your experiment wouldn’t work at all the way you expect it to?

Is that hypothesis driven experimentation or is that engineering? If your results are consistently in the ballpark on the first try, it sounds like the latter.

2 Likes

Social Constructionism feels a bit abstract at first, but it is not a jargon heavy mumbo jumbo like much of the post-modernist camp. And is certainly not nearly as silly as the priming bullshit, which is junk-science on the Malcom Gladwell level (who publishes complete crap).

Basically, the easiest way to way to understand social contrsuctionsism is that our understanding of the world is developed in relation to others, in relation to our environments, etc. Our ideas, personalities, tastes, etc. do not come strictly from within, but are things we are also things we are socialized into.

The easiest way to understand this is by thinking about what is considered “right” and “wrong” in various nations, states, families, or pairs. Is prostitution simply wrong? Is it inherently evil? Maybe here in America we believe that, but in The Netherlands it is not believed to be so. Is murder wrong? Of course! Well, that is unless it is in self-defense, or during war, etc. Is having sex with someone who isn’t your spouse wrong? Not always, just think back to the “swinging 70’s.”

There are aspects of social science that are crap (priming is a great example of this) but just because something can feel a bit abstract doesn’t mean it is jargon heavy mumbo jumbo. Social Constructionism is on pretty solid footing.

3 Likes

Like if your country has “Democratic” in the name?

2 Likes

Well-put.

And, social constructionism is more of an epistemology than a “science.” It can be applied to a variety of science and non-science domains (e.g. literature, history, anthropology). So saying “hurr, social constructionism not replicable!” is profoundly ignorant of what social constructionism even is. (Never mind that research methods related to social constructionism, e.g. qualitative research, have their own ways of addressing questions about reliability and validity - it’s so much easier to just scoff.)

1 Like

I wasn’t picking on that specifically, it was just what I was reading today. There are a lot of fields that seem to me like they could be covered by three or four really good people, and, hopefully, fifty years down the road,. maybe one curmudgeon.