The Observer's Bias Paradox (Is this really a paradox?)

Reply to rickyk95 You can control for the biases you believe to be there. And if you can control for the particular biases of some specific domain, then you can also control for bias in a general fashion when investigating the question of scientific bias itself.

So as is the general case with these kinds of self-referential paradoxes, you can break out of the apparent circularity by referencing hierarchy. Different levels of analysis - the general vs the particular - break apart the deductive loop to allow conclusions to be arrived at via inductive reasoning.

That means of course that you can't transcend the conditions for knowledge. You can't get outside the world you are trying to organise and so prove absolutely some claim - like that you have correctly removed all possible observer biases in your attempt to demonstrate that observer bias is indeed a real thing.

But you can then quite straightforwardly demonstrate that you have minimised your uncertainty about this being true. If you frame a general theory - the hypothesis that observer bias exists in science - you can then check the degree to which that prediction measures as true. A theory that is general enough should even predict the biases you will bring to the table when exploring this hypothesis.

So while you can't break out of the circle of explanation, you can show that the general and the particular - the theory and its tests - are becoming ever more definite. The more possibility the general rule about observer bias absorbs, the more unlikely it is that any of the particular forms of bias will escape notice.

Induction means accepting probabilities rather than demanding certainties. But in the end, that is how knowledge works. And science has developed a vast array of practical tools for dealing with observer bias - even if it is well known that it gets pretty relaxed about actually applying them much of the time.

Reply to rickyk95 I agree. Scientists are humans and carry all the biases of any human (economic greed, glory, power, etc.). It is fanciful thinking to believe that somehow science is policing itself. It is as subject to herd thinking and self-protection as any other profession, and the bigger the payoff, the more likely such biases will exist.

I would argue that the degree of 'observer bias' varies in direct proportion to the extent to which the subject matter falls under the heading 'social sciences'. In other words, it is less likely to occur in physics, chemistry, astronomy, and more likely to occur in psychology, sociology or political science.

I think this is also a facet of the so-called 'replication crisis', about which the Wikipedia entry says

The replication crisis refers to a methodological crisis in science in which scientists have found that the results of many scientific studies are difficult or impossible to replicate on subsequent investigation, either by independent researchers or by the original researchers themselves. While the crisis has long-standing roots, the phrase was coined in the early 2010s as part of a growing awareness of the problem.

Since the reproducibility of experiments is an essential part of the scientific method, the inability to replicate the studies of others has potentially grave consequences for many fields of science in which significant theories are grounded on unreproduceable experimental work.

The replication crisis has been particularly widely discussed in the field of psychology (and in particular, social psychology) [i.e. what I said ;-) ] and in medicine, refers to a methodological crisis in science in which scientists have found that the results of many scientific studies are difficult or impossible to replicate on subsequent investigation, either by independent researchers or by the original researchers themselves.[1] While the crisis has long-standing roots, the phrase was coined in the early 2010s as part of a growing awareness of the problem.

Since the reproducibility of experiments is an essential part of the scientific method, the inability to replicate the studies of others has potentially grave consequences for many fields of science in which significant theories are grounded on unreproduceable experimental work.

The replication crisis has been particularly widely discussed in the field of psychology (and in particular, social psychology) and in medicine, where a number of efforts have been made to re-investigate classic results, and to attempt to determine both the reliability of the results, and, if found to be unreliable, the reasons for the failure of replication

There was an important precursor to this general crisis in the findings of John Ioannidis, described in Lies, Damned Lies, and Medical Science.

Another relevant recent article is on the supposed 'PSI research' conducted by Daryl Bem which purported to provide evidence for foreknowledge, described in Daryl Bem Proved ESP Is Real (which means science is broken) is relevant to this question (long read).

Reply to Wayfarer I would argue bias and outright fraud it's most likely to b be found in the medical industry where the money is beyond imaginable and growing leaps and bounds.


https://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/308269/

"He and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies—conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain—is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed. '

Reply to rickyk95 It's interesting. The question is "How does an error-prone instrument detect error?"

I think the observer bias (OB) is human error (HE). The data itself is pristine. And the OB is specific to a particular study/experiment. That mean anyone who doesn't have anything to gain/lose from the study/experiment is free of the OB. Perhaps it's these independent observers who detected the observer bias. Of course there can't be an observer who isn't biased but the point is their biases are irrelevant to the study/experiment. For instance the sexual preferences (bias) of an observer has no relevance/effect to/on a scientific experiment.

Does that answer your question?

We should all beware of belittling science however - note this article in today's media http://www.slate.com/articles/health_and_science/science/2017/08/science_is_not_self_correcting_science_is_broken.html

I detest the 'anti-science' of the alt right and US creationism. Science Is absolutely indispensable. What it is not, is a guide to ethics; scientists may be good people, but being a good person doesn't depend on them being scientists. If they're good people, they will do everything they can to eliminate confirmation bias; and this will make them good scientists as well.

Reply to Wayfarer Unfounded idolatry. They are what they are, and human like everyone else.

Quoting rickyk95

Scenario 1: This only proves that there is observer bias even further, because the researchers that discovered the observers bias suffered from it themselves.

Scenario 2: It doesnt, because the arguments that proved or came about suggesting that an observer's bias exists were flawed (due to the bias), so it takes all of the findings' validity.

Is this just me or is there something here?

There's something more. It's called peer review. Not everyone has the same biases. One scientist can announce their hypothesis, but it isn't doesn't change from a hypothesis to a theory until other scientists in the same field run the same experiments and get the same results. This is what makes scientific theories objective as opposed to the subjective hypotheses of say, religion. The problem you describe is more of a problem for religion than it is for science.

I think recognizing the different variables in experiments help to limit biases as well.

Quoting Harry Hindu

There's something more. It's called peer review.

It's all biased. Of course the process of peer review and its inherent biases have been brought into consideration. There are all types of biases flowing through science as it does every other part of our society. The Dredd Scott decision was peer reviewed and look what judicial peer review brought us.

Quoting Rich

The Dredd Scott decision

Hmmmmmm.

Reply to Srap Tasmaner The issue is there is no such thing as a peer review that is not subject to prejudice, economic influences, biasses, and outright fraud.

Opioid studies were subjected to all kinds of peer reviews, creating a sense among the less cynical population that all is well. So far, over 150,000 prescription users have been killed and who knows what shape the other users are in (presumably a bit better than those who are dead).

Bottom line, peer review means nothing when big money is at stake. Check out the history of talc that is now receiving lots of press because of yesterday's $400 million judgement and the role that science and government plays in the industry.

Quoting Wayfarer

I would argue that the degree of 'observer bias' varies in direct proportion to the extent to which the subject matter falls under the heading 'social sciences'. In other words, it is less likely to occur in physics, chemistry, astronomy, and more likely to occur in psychology, sociology or political science.

My caveat to this would be that much reputable social science understands this in advance and the researcher(s) outline their biasses upfront. You can't study political views and not have a political position. Which would be a good reason for remembering economics is really political economy.

As against this, medics aren't always clear about their own biasses. Andrew Carr at Oxford has been doing interesting work over the last few years, for instance, about the placebo effect of surgery. Much minor surgery - e.g. knee arthroscopy, gastric balloons - seems to have no net benefit over the placebo effect.

Quoting Rich

Unfounded idolatry....

Do you think that anti-vaccination, climate change denial, and young-earth creationism are valid forms of scepticism with respect to science?

Quoting mcdoodle

My caveat to this would be that much reputable social science understands this in advance and the researcher(s) outline their biasses upfront.

Sometimes hard to look at your spectacles, rather than through them. A case in point - there was a story here the other day about a massive trial of 'genetic factors in depression' (see this story.) It wants tens of thousands of people to volunteer so as to 'crack the genetic code' so as to develop better drugs. But I am sceptical that depression has a purely genetic or bio-organic basis; large-scale depression might be caused by the disintegration of traditional social structures, or the rate of change, or inability to adapt to the modern world, or any number of other factors. But the medical and genetic approaches seem to offer the most concrete types of answers to such poorly understood problems and besides attract the attention of "Big Pharma" as noted above.

They're the kinds of cultural biases I have in mind when I think about this problem.

Quoting Wayfarer

Do you think that anti-vaccination, climate change denial, and young-earth creationism are valid forms of scepticism with respect to science?

Don't trust and definitely verify.

Observe the numbers of people being killed, sit back and meditate on the numbers. They are astronomical.

Reply to Rich You're not making sense.

Reply to Wayfarer I am to those who have studied the industry and how it works with government. Just what one would expect if they understand human nature and history.