Reproducibility in Science

That they're attempting to diagnose the problem by surveying scientists seems to be evidence of the underlying problem. This is science after all, and it should be obvious to a scientist that its entirely irrelevant whether scientists believe there to be a crisis or not. If we define a scientific result as that which is reproducible, and we have evidence that it's statistically unlikely that most published findings are reproducible, then, as a matter of fact, journal articles are not to be considered a trusted source of scientific information. Whether scientists wish to gather together and declare the current state a crisis isn't a scientific event; it's a political one.

The solution is to qualify all scientific research for what it is. If it's preliminary and based only upon what one lab has determined, then it should be noted. If it has been reproduced, then our confidence level in those results can be increased. The political question then becomes what everyone wishes to do with the preliminary results, especially if there are no reproduced findings forthcoming. If we learn that talcum powder, for example, causes cancer based upon a single lab, do we shut down that entire industry or not?

Quoting Hanover

it should be obvious to a scientist that its entirely irrelevant whether scientists believe there to be a crisis or not.

Good point.

Quoting SystemsActivated

Any thoughts on this series?

According to Nature, there's a good Wittgensteinian preliminary answer to this: that 'reproducibility' is a mistakenly catch-all term, which actually embraces three sorts of reproducibles: methods, results, and inference.

http://www.nature.com/news/muddled-meanings-hamper-efforts-to-fix-reproducibility-crisis-1.20076

Even when we've done that, though, we're faced with the general difficulty: how does the mythology of science stand up to scrutiny if it has to forgo a simple myth of experimental reproducibility? Today I watched with a friend the NASA reports back from Juno: my friend had participated in a small way in the project. Marvellous! We can see the moons! Stuff like that just has to work, or not, as the case may be: where you're about to orbit Jupiter five years from home there's no second thoughts. That is to echo what Hanover is implying: that we have to have criteria for what findings we act upon, quality control of a much higher order than is prevalent.

Here there has to be greater openness in medical research: we've had thirty years of anti-depressants for instance based on quite a weak correlation.

Psychology and neuroscience seem to me particularly vulnerable. The lack of an underlying philosophical basis reinforces the mootness of many supposed findings in these fields; neuroscience especially has already come under attack for small samples and weak statistical methodology.

But, as BC and Hanover say, there's only a crisis if the wider world than the scientific community think there's a crisis, and this depends on PR as much as rational scrutiny.

"Science" has gotten much, much bigger than it used to be. First, more phenomena are being subjected to granular examination that previously. Second, the vaguely defined enterprise of "science" has gotten much bigger than previously (say, 50 years ago). Third, "scientific" as a positive adjective has been appropriated by many fields where it may, or may not be appropriate.

All this adds up to a lot more people engaged in real science on one end of the spectrum and outright fraud and deception at the other end. In between are sciency, researchy, and laboratoryish activities. It's not all that hard to put together a researchy, sciency report claiming that dogs don't like to be hugged. Get some stray dogs, wire them up, and sequentially squeeze them (which, from a dog's POV might look more like the preliminaries of some ghastly procedure) and presto, results. "Dogs don't like to be hugged."

Something that I didn't see mentioned in the three articles (full disclosure: I gave them a cursory read) was that a lot of publicly funded research doesn't get published at all. Researchers are loathe to report research that reveals no progress, or negative results. You treat 1000 patients with XYZ1234 drug and nothing happens. Why report it? Well, "no results" is a result. If all research that was publicly paid for had to be published (a lot of it isn't) we might find that we either paying for more bad science than we thought, some research grantors respond more to hype than to heft, or that there are more blind alleys than we thought. But we can't know any of this if researchers don't publish results.

Quoting mcdoodle

we've had thirty years of anti-depressants for instance based on quite a weak correlatio

It's worse than weak correlation. Assuming a lab has a unique chemical, and not just a chemical different enough from other antidepressants to count as new, tests begin. First, chemical analysis. Then tests on animals to determine safety. Eventually we get around to human safety trials -- can the drug be swallowed safely, whether it works or not. Finally, the trials to see whether it works--does it reduce the symptoms of depression?

The number of subjects in the trial is often much smaller than you would think necessary: a few dozen, a couple hundred. Small numbers.

Maybe there will be another small trial or two, and the drug will receive FDA approval. Next, the advertising: "Ask your doctor for La-La-Zone™ for those free-floating feelings of hostility, barely controlled rage, and existential despair." A million people take it over the next two years, and eventually all the reports of adverse outcomes have trickled in, and it is found that LaLaZone™ can result in terroristic ideation in 1% of the cases, especially among people likely to vote for far-right wing candidates, in whatever country they happen to reside. Those are the people, of course, closest to acting on their free-floating hostilities, murderous rages, and existential agonies. Now we get a black box warning for the drug: Be careful of who you prescribe it too.

The drug is quite profitable, so marketing continues, black box warning and all, and lots of people keep taking the drug. Some are helped, some end up being shot during the expression of free-floating murderous rage, and some just stop taking the drug and the world keeps on spinning.

LaLaZone is a bit of an exaggeration, but not a wild exaggeration. Trial cohorts are small, and the people who buy the drug are the real lab rats. The neurochemistry of mood and modification isn't all that well understood, and Koch's Postulates don't work for mental distress. There's usually no "specific agent", no lesion, no intervention that can be precisely tracked.