Peer review at work

[LogicalFallacy]

https://www.livescience.com/62832-stanford-prison-experiment-flawed.html?utm_source=notification 

 

Some may point to the above article and say ah ha, see science doesn't work. But I'd posit, that though it took some decades, this is science working. Discovering what is true and what isn't Certainly it means we should be skeptical, and never hold onto a position absolutely, but it doesn't mean we should or need to be cynical. 

 

That's my random musings right now.

[Joshpantera]

I'm not surprised at this. 

[Orbit]

In sociology, we use that study as an example of how NOT to do research, so no harm done from my perspective. And yes, science is self-correcting.

[midniterider]

9 hours ago, LogicalFallacy said:

https://www.livescience.com/62832-stanford-prison-experiment-flawed.html?utm_source=notification 

 

Some may point to the above article and say ah ha, see science doesn't work. But I'd posit, that though it took some decades, this is science working. Discovering what is true and what isn't Certainly it means we should be skeptical, and never hold onto a position absolutely, but it doesn't mean we should or need to be cynical. 

 

That's my random musings right now.

 

Holding onto a position as absolute is not the pursuit of science, but the pursuit of a dogmatic belief system. I'm glad you mentioned that.

 

To say that science doesn't work would be quite a generalization in a very broad field. A failed social experiment has zero impact on design and production of microprocessors and antibiotics. We may want to revisit how we do social experiments though.

[pantheory]

Psychology, sociology and a number of other sciences can be called soft sciences. Results of experiments and interpretations are often debatable. Unfortunately the same kinds of problems and misinterpretations, but to a lesser extent, can also happen in hard sciences such as particle physics, cosmology, quantum mechanics, as well as nearly all theory-based research and studies. Experiments and observations can be part of biased studies producing wrong conclusions backed by wrongly structured statistics.  Science is a methodology directed towards our best efforts to find objective answers, but sometimes it goes awry. IMO in the long run science advances much more than it makes mistakes, maybe three steps forward and one step backwards concerning theory -- on a long term basis.

[Bhim]

Heh, so we're calling psychology a science now?  Mostly kidding, and before I offend Orbit I'll move on.

 

I read about this earlier this week. Indeed I too learned about this experiment in high school sociology, and was surprised to see such a foundational experiment overturned. Is this peer-review at work? Let me start with a controversial comment. Imagine this wasn't a seminal study in psychology, but in climate science. What if we determined that the anthropogenic contribution to climate change is in fact negligible? Well, that would be a problem given said study would have been responsible for carbon emission standards, international agreements, and a fundamental industrial shift from fossil fuels to renewable energy sources. There would rightly be clamor over the fact that the earth's (political) axis was shifted on the basis of scientific work whose results were fundamentally overturned by later research.

 

Now, my point here isn't that climate science is flawed. To be clear, it isn't. The point I hope to make is that while science is naturally self-correcting, it is also meant to converge on the truth. Later theories should refine earlier theories rather than invalidate them. A famous example of this is the General Theory of Relativity, which makes corrections to Newtonian gravity rather than declaring it obsolete. So if the prison experiment were done properly, then later experiments would perhaps show that the results do not hold in certain circumstances or when certain types of people are participants. But here we have gross experimental errors. One participant faked a breakdown to get out of the experiment early. The PI changed his experimental protocol because of a suggestion from his girlfriend. These aren't the sorts of things that are meant to be corrected by peer review. These are things that simply shouldn't happen. It's akin to a climate researcher adding the number 6 to every temperature measurement he records. From what I'm reading here, it's simply science done poorly.

 

Scientific findings are always subject to dispute and refinement. But if the science is done correctly, successive findings should converge rather than contradict. The widespread non-reproducible nature of findings in the social sciences makes me worry about the veracity of this sort of research. Thank goodness social scientists are at least well-versed in statistics.

[wellnamed]

Some articles on flaws in research that I've enjoyed reading:

 

How a subtle mistake in a poli-sci paper became headline news

 

P-hack your way to scientific glory

 

An Erratum on trends in men's facial hair

 

Also Howard Becker's recent book Evidence is really good. I especially like the way he discusses the relationships between data, evidence, and ideas in scientific work. He has a way of getting to the heart of the difficulties with research very clearly and accessibly.

 

I feel like a lot of the time the actual process of science is pretty obscure from reading news articles about studies. It becomes a lot more meaningful when you are able to start reading articles themselves and thinking about the methods used and their limitations. I also like that, in the case of the facial hair study, the authors included the raw data as an appendix to the original article, which is what allowed others to find and point out the error. Especially now with the internet making it very easy to share even large datasets, I think all scientists should try to make all the data necessary to reproduce their results/calculations available.

[Orbit]

Just ran across this in the Chronicle: https://www.chronicle.com/article/The-3-Types-of-Peer-Reviewers/243698

 

And Bhim, it's called social science for a reason.

[wellnamed]

1 minute ago, Orbit said:

And Bhim, it's called social science for a reason. 

 

Is it because those are the folks you go to when you need a socialist professor?

[Orbit]

Just now, wellnamed said:

 

Is it because those are the folks you go to when you need a socialist professor?

 

Actual quote from a Criminal Justice professor to me, in a Freudian slip "I've got a student here who needs to talk to a socialist....I mean a sociologist"

[florduh]

https://www.psychologytoday.com/us/blog/theory-knowledge/201601/the-is-psychology-science-debate

[Orbit]

Psychology always seemed to be a strange hybrid to me, because of the focus on counseling and trying to explain what goes on in people's subconsciousness. But then again, I may not know enough about what research psychologists do beyond running rats through mazes and using undergrads as research guinea pigs.

[Bhim]

4 hours ago, Orbit said:

Psychology always seemed to be a strange hybrid to me, because of the focus on counseling and trying to explain what goes on in people's subconsciousness. But then again, I may not know enough about what research psychologists do beyond running rats through mazes and using undergrads as research guinea pigs.

 

I suppose it could be likened to medical science, which has both academic and clinical sides. Granted, there the focus is definitely more heavily shifted towards the clinical, but there is an element of research for its own sake.

[Orbit]

Zimbardo responds:  http://www.prisonexp.org/response