gyrofry
gyrofry
Kurdt
Post rating:
~
Beware the creeping cracks of bias
Evidence is mounting that research is riddled with systematic errors. Left unchecked, this could erode public trust, warns Daniel Sarewitz.
09 May 2012
Alarming cracks are starting to penetrate deep into the scientific edifice. They threaten the status of science and its value to society. And they cannot be blamed on the usual suspects — inadequate funding, misconduct, political interference, an illiterate public. Their cause is bias, and the threat they pose goes to the heart of research.
Bias is an inescapable element of research, especially in fields such as biomedicine that strive to isolate cause–effect relations in complex systems in which relevant variables and phenomena can never be fully identified or characterized. Yet if biases were random, then multiple studies ought to converge on truth. Evidence is mounting that biases are not random. A Comment in Nature in March reported that researchers at Amgen were able to confirm the results of only six of 53 'landmark studies' in preclinical cancer research (C. G. Begley & L. M. Ellis Nature 483, 531–533; 2012). For more than a decade, and with increasing frequency, scientists and journalists have pointed out similar problems.
Early signs of trouble were appearing by the mid-1990s, when researchers began to document systematic positive bias in clinical trials funded by the pharmaceutical industry. Initially these biases seemed easy to address, and in some ways they offered psychological comfort. The problem, after all, was not with science, but with the poison of the profit motive. It could be countered with strict requirements to disclose conflicts of interest and to report all clinical trials.
Yet closer examination showed that the trouble ran deeper. Science's internal controls on bias were failing, and bias and error were trending in the same direction — towards the pervasive over-selection and over-reporting of false positive results. The problem was most provocatively asserted in a now-famous 2005 paper by John Ioannidis, currently at Stanford University in California: 'Why Most Published Research Findings Are False' (J. P. A. Ioannidis PLoS Med. 2, e124; 2005). Evidence of systematic positive bias was turning up in research ranging from basic to clinical, and on subjects ranging from genetic disease markers to testing of traditional Chinese medical practices.
How can we explain such pervasive bias? Like a magnetic field that pulls iron filings into alignment, a powerful cultural belief is aligning multiple sources of scientific bias in the same direction. The belief is that progress in science means the continual production of positive findings. All involved benefit from positive results, and from the appearance of progress. Scientists are rewarded both intellectually and professionally, science administrators are empowered and the public desire for a better world is answered. The lack of incentives to report negative results, replicate experiments or recognize inconsistencies, ambiguities and uncertainties is widely appreciated — but the necessary cultural change is incredibly difficult to achieve.
Researchers seek to reduce bias through tightly controlled experimental investigations. In doing so, however, they are also moving farther away from the real-world complexity in which scientific results must be applied to solve problems. The consequences of this strategy have become acutely apparent in mouse-model research. The technology to produce unlimited numbers of identical transgenic mice attracts legions of researchers and abundant funding because it allows for controlled, replicable experiments and rigorous hypothesis-testing — the canonical tenets of 'scientific excellence'. But the findings of such research often turn out to be invalid when applied to humans.
A biased scientific result is no different from a useless one. Neither can be turned into a real-world application. So it is not surprising that the cracks in the edifice are showing up first in the biomedical realm, because research results are constantly put to the practical test of improving human health. Nor is it surprising, even if it is painfully ironic, that some of the most troubling research to document these problems has come from industry, precisely because industry's profits depend on the results of basic biomedical science to help guide drug-development choices.
Scientists rightly extol the capacity of research to self-correct. But the lesson coming from biomedicine is that this self-correction depends not just on competition between researchers, but also on the close ties between science and its application that allow society to push back against biased and useless results.
It would therefore be naive to believe that systematic error is a problem for biomedicine alone. It is likely to be prevalent in any field that seeks to predict the behaviour of complex systems — economics, ecology, environmental science, epidemiology and so on. The cracks will be there, they are just harder to spot because it is harder to test research results through direct technological applications (such as drugs) and straightforward indicators of desired outcomes (such as reduced morbidity and mortality).
Nothing will corrode public trust more than a creeping awareness that scientists are unable to live up to the standards that they have set for themselves. Useful steps to deal with this threat may range from reducing the hype from universities and journals about specific projects, to strengthening collaborations between those involved in fundamental research and those who will put the results to use in the real world. There are no easy solutions. The first step is to face up to the problem — before the cracks undermine the very foundations of science.
Daniel Sarewitz is co-director of the Consortium for Science, Policy and Outcomes at Arizona State University, and is based in Washington DC.
http://www.nature.com/news/beware-the-creeping-cracks-of-bias-1.10600
Evidence is mounting that research is riddled with systematic errors. Left unchecked, this could erode public trust, warns Daniel Sarewitz.
09 May 2012
Alarming cracks are starting to penetrate deep into the scientific edifice. They threaten the status of science and its value to society. And they cannot be blamed on the usual suspects — inadequate funding, misconduct, political interference, an illiterate public. Their cause is bias, and the threat they pose goes to the heart of research.
Bias is an inescapable element of research, especially in fields such as biomedicine that strive to isolate cause–effect relations in complex systems in which relevant variables and phenomena can never be fully identified or characterized. Yet if biases were random, then multiple studies ought to converge on truth. Evidence is mounting that biases are not random. A Comment in Nature in March reported that researchers at Amgen were able to confirm the results of only six of 53 'landmark studies' in preclinical cancer research (C. G. Begley & L. M. Ellis Nature 483, 531–533; 2012). For more than a decade, and with increasing frequency, scientists and journalists have pointed out similar problems.
Early signs of trouble were appearing by the mid-1990s, when researchers began to document systematic positive bias in clinical trials funded by the pharmaceutical industry. Initially these biases seemed easy to address, and in some ways they offered psychological comfort. The problem, after all, was not with science, but with the poison of the profit motive. It could be countered with strict requirements to disclose conflicts of interest and to report all clinical trials.
Yet closer examination showed that the trouble ran deeper. Science's internal controls on bias were failing, and bias and error were trending in the same direction — towards the pervasive over-selection and over-reporting of false positive results. The problem was most provocatively asserted in a now-famous 2005 paper by John Ioannidis, currently at Stanford University in California: 'Why Most Published Research Findings Are False' (J. P. A. Ioannidis PLoS Med. 2, e124; 2005). Evidence of systematic positive bias was turning up in research ranging from basic to clinical, and on subjects ranging from genetic disease markers to testing of traditional Chinese medical practices.
How can we explain such pervasive bias? Like a magnetic field that pulls iron filings into alignment, a powerful cultural belief is aligning multiple sources of scientific bias in the same direction. The belief is that progress in science means the continual production of positive findings. All involved benefit from positive results, and from the appearance of progress. Scientists are rewarded both intellectually and professionally, science administrators are empowered and the public desire for a better world is answered. The lack of incentives to report negative results, replicate experiments or recognize inconsistencies, ambiguities and uncertainties is widely appreciated — but the necessary cultural change is incredibly difficult to achieve.
Researchers seek to reduce bias through tightly controlled experimental investigations. In doing so, however, they are also moving farther away from the real-world complexity in which scientific results must be applied to solve problems. The consequences of this strategy have become acutely apparent in mouse-model research. The technology to produce unlimited numbers of identical transgenic mice attracts legions of researchers and abundant funding because it allows for controlled, replicable experiments and rigorous hypothesis-testing — the canonical tenets of 'scientific excellence'. But the findings of such research often turn out to be invalid when applied to humans.
A biased scientific result is no different from a useless one. Neither can be turned into a real-world application. So it is not surprising that the cracks in the edifice are showing up first in the biomedical realm, because research results are constantly put to the practical test of improving human health. Nor is it surprising, even if it is painfully ironic, that some of the most troubling research to document these problems has come from industry, precisely because industry's profits depend on the results of basic biomedical science to help guide drug-development choices.
Scientists rightly extol the capacity of research to self-correct. But the lesson coming from biomedicine is that this self-correction depends not just on competition between researchers, but also on the close ties between science and its application that allow society to push back against biased and useless results.
It would therefore be naive to believe that systematic error is a problem for biomedicine alone. It is likely to be prevalent in any field that seeks to predict the behaviour of complex systems — economics, ecology, environmental science, epidemiology and so on. The cracks will be there, they are just harder to spot because it is harder to test research results through direct technological applications (such as drugs) and straightforward indicators of desired outcomes (such as reduced morbidity and mortality).
Nothing will corrode public trust more than a creeping awareness that scientists are unable to live up to the standards that they have set for themselves. Useful steps to deal with this threat may range from reducing the hype from universities and journals about specific projects, to strengthening collaborations between those involved in fundamental research and those who will put the results to use in the real world. There are no easy solutions. The first step is to face up to the problem — before the cracks undermine the very foundations of science.
Daniel Sarewitz is co-director of the Consortium for Science, Policy and Outcomes at Arizona State University, and is based in Washington DC.
http://www.nature.com/news/beware-the-creeping-cracks-of-bias-1.10600
[account deactivated]
GoldenLionTamarin
GoldenLionTamarin
Goldgelbes Löwenäffchen
Post rating:
~
if science is bullshit then why should i liste nto a scientist. your move, buddy
dipshit420
dipshit420
political car tune
Post rating:
~
Obamas libsquad at it again
girdles_gone_wild
girdles_gone_wild
Post rating:
~
post this on the ronald reagan forums
guidoanselmi
guidoanselmi
meet me in st louis
Post rating:
~
tpaine posted:Fucking Science, Again!
~my job description~
Keven
Keven
Post rating:
+
Problem: science is riddled with bias
Solution: oh well im not gonna read that but i assume it's referring to work that i, an engineer student, don't understand or like such as ethnography or economics. Probably the fix is to make things more double blind, another phrase i don't understand and never will. Thanks for reading.
Solution: oh well im not gonna read that but i assume it's referring to work that i, an engineer student, don't understand or like such as ethnography or economics. Probably the fix is to make things more double blind, another phrase i don't understand and never will. Thanks for reading.
swampman
swampman
Dont Mess With Sucess
Post rating:
~
Why cant we wait to deal with this issue until we mining asteroids
shennong
shennong
Post rating:
~
the funny thing is that Ioannidis' work was really well received and his group is highly respected but no one has any idea what to do about it so we all just kind of ignore it lol
cleanhands
cleanhands
Post rating:
~
man im sure glad i studied natural sciences instead of some mickey mouse bullshit like sociology or economics *is economically coerced into participating in a cargo cult*
Doug
Doug
Post rating:
~
peer review sucks!!
Empathy
Empathy
Post rating:
~
Peer review aka cabal approval
getfiscal
getfiscal
Mike Nelson here.
Post rating:
+
peer review sucks because these anonymous cowards won't sign off on my research that suggests diet coke can turn you into an immortal
Doug
Doug
Post rating:
~
p values also suck!
Ironicwarcriminal
Ironicwarcriminal
Post rating:
~
what do you expect from a group of people who make up fraudulent "death threats" against themselves to get pity and give themselves a persecution complex
apparently this is a "Death threat" that forced ANU scientists to be moved to "secure premises"
qqqqqqq
apparently this is a "Death threat" that forced ANU scientists to be moved to "secure premises"
qqqqqqq
Edited by Ironicwarcriminal ()
[account deactivated]
guidoanselmi
guidoanselmi
meet me in st louis
Post rating:
~
shirley...
germanjoey
germanjoey
Post rating:
~
Doug posted:peer review sucks!!
Now listen to my song!
[account deactivated]