Problems in medical research:Tracking outcome switching

For the discussion of the sciences. Physics problems, chemistry equations, biology weirdness, it all goes here.

Moderators: gmalivuk, Moderators General, Prelates

HungryHobo
Posts: 1708
Joined: Wed Oct 20, 2010 9:01 am UTC

Problems in medical research:Tracking outcome switching

Postby HungryHobo » Sun Feb 07, 2016 11:19 am UTC

http://compare-trials.org/

Ben Goldacre (author of the book Bad Science) and a team recently set up a website for tracking improperly switched/unreported/added outcomes in clinical trials.

Outcome switching in clinical trials is a serious problem. Between October 2015 and January 2016, the COMPare team systematically checked every trial published in the top five medical journals, to see if they misreported their findings:

We compared each clinical trial report with its protocol or registry entry. Some trials reported their outcomes perfectly. For the others, we counted how many of the outcomes pre-specified in the protocol or registry were never reported. We also counted how many new outcomes were silently added.

When we detected unreported or added outcomes, we wrote a letter to the journal pointing them out. We tracked which journals published our letters – and which did not.


The reactions of some (supposedly respectable) journals has been surprisingly poor while others have published corrections.

So far of 67 trials checked 9 were perfect, and in the remaining 58 trials a total of 301 prespecified outcomes were silently not reported and 357 were silently added .

58 letters have been sent to journals about switched outcomes of which only 6 have been published.

It is a major problem in modern clinical trial publishing since outcome switching destroys statistical validity.

Part of the challenge of getting public support for fixing the problem is that in order to understand why it's such a big problem people have to wade through a great deal of tedium.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.

Frenetic Pony
Posts: 177
Joined: Fri Jan 27, 2012 6:31 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby Frenetic Pony » Mon Feb 08, 2016 1:35 am UTC

Publish (positive outcomes) or Perish. This kind of thing happens a lot unless it's in testable physics (in which case, well, you can test it quite accurately, and people most certainly will).

Just allowing and showing off the publishing of outcomes not in line with the thesis would be helpful both to careers and to scientific knowledge in general.

User avatar
Izawwlgood
WINNING
Posts: 18686
Joined: Mon Nov 19, 2007 3:55 pm UTC
Location: There may be lovelier lovelies...

Re: Problems in medical research:Tracking outcome switching

Postby Izawwlgood » Mon Feb 08, 2016 1:54 am UTC

So, the NIH requires that all clinical trials that involve human subjects be reported to clinicaltrials.org. This has been going on since about 2007 or so. That said, yes, of course, there's lot of shitty reporting of science because journals only publish positive results.
... with gigantic melancholies and gigantic mirth, to tread the jeweled thrones of the Earth under his sandalled feet.

p1t1o
Posts: 960
Joined: Wed Nov 10, 2010 4:32 pm UTC
Location: London, UK

Re: Problems in medical research:Tracking outcome switching

Postby p1t1o » Wed Feb 10, 2016 4:31 pm UTC

I've done some work on some minor medical devices and was involved on the periphery of some of the clinical trials analysis.
What I saw was that, firstly, not all clinical trials are for life-saving anti-cancer drugs or things like that. A lot of them are for fairly minor things.

Secondly, I saw how rarely one gets exactly the result you want from a study. Once you have collected all the data you can, and you have not achieved your endpoint, you are left trying to find something that you CAN take away from the study, sometimes this means adding/removing outcomes from your analysis.

The fact that clinical trials are lengthy and expensive puts a great deal of pressure on the project to at least provide SOME kind of useful data. Unless you ARE working on a super-life-saving drug for huge pharma, you will never get a chance to run another trial, you only get one shot at your data.

So I would say that though the numbers that the OP presents are surprising, each modification does not necessarily represent a malign attempt to pervert science or corruption of any other sort. Though inevitably some of them will I suppose.

I would be very interested if the organisation behind compare-trials.org looked into what the data would have looked like otherwise.

HungryHobo
Posts: 1708
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby HungryHobo » Sun Feb 14, 2016 9:20 am UTC

p1t1o wrote:So I would say that though the numbers that the OP presents are surprising, each modification does not necessarily represent a malign attempt to pervert science or corruption of any other sort.


Thing is, it's perfectly acceptable to add interesting findings but it's important to state clearly that they are incidental findings rather than part of the pre-reg plan.

The strong incentive to p-hack and data-dredge is part of the problem.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.

mfb
Posts: 950
Joined: Thu Jan 08, 2009 7:48 pm UTC

Re: Problems in medical research:Tracking outcome switching

Postby mfb » Sun Feb 14, 2016 11:13 am UTC

p1t1o wrote:The fact that clinical trials are lengthy and expensive puts a great deal of pressure on the project to at least provide SOME kind of useful data.
A null result is useful.

I could write a lengthy post, but I think that is the main point. As long as null results are not considered as equally valid and useful as "significant" results (come on, p<0.05 is a joke), medical publications will stay massively biased.

lorb
Posts: 405
Joined: Wed Nov 10, 2010 10:34 am UTC
Location: Austria

Re: Problems in medical research:Tracking outcome switching

Postby lorb » Mon Feb 15, 2016 10:54 am UTC

mfb wrote:
p1t1o wrote:The fact that clinical trials are lengthy and expensive puts a great deal of pressure on the project to at least provide SOME kind of useful data.
A null result is useful.

I could write a lengthy post, but I think that is the main point. As long as null results are not considered as equally valid and useful as "significant" results (come on, p<0.05 is a joke), medical publications will stay massively biased.


Outside of a purely academic context null results are in most cases not equally useful. Finding that drug X can cure Y is much more useful than finding that it can't.
Please be gracious in judging my english. (I am not a native speaker/writer.)
http://decodedarfur.org/

mfb
Posts: 950
Joined: Thu Jan 08, 2009 7:48 pm UTC

Re: Problems in medical research:Tracking outcome switching

Postby mfb » Tue Feb 16, 2016 12:20 am UTC

lorb wrote:Outside of a purely academic context null results are in most cases not equally useful. Finding that drug X can cure Y is much more useful than finding that it can't.
It is a more desirable result, but the use of the study is the test of the drug. If it is positive: great. If it does not have a measurable effect, you know that you can stop testing (or repeat with larger test samples if even a small effect would be interesting), which is also relevant.

HungryHobo
Posts: 1708
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby HungryHobo » Tue Feb 16, 2016 1:58 am UTC

lorb wrote:Outside of a purely academic context null results are in most cases not equally useful. Finding that drug X can cure Y is much more useful than finding that it can't.


Ask a doctor: "do you want to know if a drug does not work?"

Indeed finding out that a drug doesn't work on children or women or people from certain populations is highly valuable information.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.

lorb
Posts: 405
Joined: Wed Nov 10, 2010 10:34 am UTC
Location: Austria

Re: Problems in medical research:Tracking outcome switching

Postby lorb » Tue Feb 16, 2016 8:05 am UTC

While I totally agree that a null result is valuable information, science does not happen in an academic vacuum. Try to think about it from an economic perspective for example. In most cases a positive result has a far better likelihood to lead to some way to make money from it. In the eyes of the party that finances it the use of a study is not to test a drug but to generate an income stream.
Please be gracious in judging my english. (I am not a native speaker/writer.)
http://decodedarfur.org/

elasto
Posts: 3778
Joined: Mon May 10, 2010 1:53 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby elasto » Tue Feb 16, 2016 8:26 am UTC

lorb wrote:While I totally agree that a null result is valuable information, science does not happen in an academic vacuum. Try to think about it from an economic perspective for example. In most cases a positive result has a far better likelihood to lead to some way to make money from it. In the eyes of the party that finances it the use of a study is not to test a drug but to generate an income stream.

And that's why the free market is not always the best - or even the most efficient - way for society to have things be managed. Research is one such place where markets fail and fail badly, I believe.

Even ignoring the fact that the market will pile research money into first-world, rich, white-man diseases because of the economic returns (when spending money elsewhere could produce much greater social returns), think about the perverse incentive for private companies to bury null results: You research if drug X deals with problem Y; You find no evidence that it does and quickly and quietly kill the study. By not publishing but keeping the information to yourself, you increase the chances that rival firms will waste time and resources on a similar study - perhaps over and over again into the future.

Very inefficient.

User avatar
Izawwlgood
WINNING
Posts: 18686
Joined: Mon Nov 19, 2007 3:55 pm UTC
Location: There may be lovelier lovelies...

Re: Problems in medical research:Tracking outcome switching

Postby Izawwlgood » Tue Feb 16, 2016 2:08 pm UTC

But that's precisely why negative results should be published, or at least made available to the public - my head spins whenever I think about the amount of public money wasted repeating experiments with negative results outcomes.

When you factor in the results of multiple trials with different results, selective publication of results suddenly becomes a very serious and very real issue.
... with gigantic melancholies and gigantic mirth, to tread the jeweled thrones of the Earth under his sandalled feet.

mfb
Posts: 950
Joined: Thu Jan 08, 2009 7:48 pm UTC

Re: Problems in medical research:Tracking outcome switching

Postby mfb » Tue Feb 16, 2016 10:33 pm UTC

lorb wrote:While I totally agree that a null result is valuable information, science does not happen in an academic vacuum. Try to think about it from an economic perspective for example. In most cases a positive result has a far better likelihood to lead to some way to make money from it. In the eyes of the party that finances it the use of a study is not to test a drug but to generate an income stream.
At the time you have the null result, the money for the study has been spent already. Publishing it is quite cheap in comparison.
Hiding a null result is scientific fraud. And also financial fraud (with potentially lethal result) if a drug is sold despite evidence that it does not work.

lorb
Posts: 405
Joined: Wed Nov 10, 2010 10:34 am UTC
Location: Austria

Re: Problems in medical research:Tracking outcome switching

Postby lorb » Tue Feb 16, 2016 11:03 pm UTC

Absolutely. Yet the way the system works there is so much incentive to get a significant result that things like outcome switching have become a widespread problem. And that is because there are many contexts in which a result that is not null is (at least perceived to be) much more useful. Also if you can make your study look like you got a significant result you are much more likely to get more funding in the future.
Please be gracious in judging my english. (I am not a native speaker/writer.)
http://decodedarfur.org/

User avatar
Izawwlgood
WINNING
Posts: 18686
Joined: Mon Nov 19, 2007 3:55 pm UTC
Location: There may be lovelier lovelies...

Re: Problems in medical research:Tracking outcome switching

Postby Izawwlgood » Wed Feb 17, 2016 4:58 am UTC

mfb wrote:Publishing it is quite cheap in comparison.
'Cheap in comparison' does not remotely mean 'cheap'.

mfb wrote:Hiding a null result is scientific fraud
This is also not true, and happens all the time in science.

mfb wrote:And also financial fraud (with potentially lethal result) if a drug is sold despite evidence that it does not work.
I don't think this is true either. Like... at all.
... with gigantic melancholies and gigantic mirth, to tread the jeweled thrones of the Earth under his sandalled feet.

mfb
Posts: 950
Joined: Thu Jan 08, 2009 7:48 pm UTC

Re: Problems in medical research:Tracking outcome switching

Postby mfb » Wed Feb 17, 2016 11:53 pm UTC

Izawwlgood wrote:'Cheap in comparison' does not remotely mean 'cheap'.
Yes, but it is like throwing a new car away because you would have to buy a steering wheel to use it. If you are fine with "just making it public" instead of a peer-reviewed publication, it is even cheaper (just the costs of writing the results down).
Izawwlgood wrote:
mfb wrote:Hiding a null result is scientific fraud
This is also not true, and happens all the time in science.
The fact that it happens frequently in some scientific disciplines doesn't make it better. It does not happen in particle physics, for example.
Izawwlgood wrote:
mfb wrote:And also financial fraud (with potentially lethal result) if a drug is sold despite evidence that it does not work.
I don't think this is true either. Like... at all.
"Use this drug, we have 2 studies that show its effect (p<0.05)" and 50 studies that found no effect. Also, it will stop you using a drug that actually does work.
Not at all?

User avatar
Izawwlgood
WINNING
Posts: 18686
Joined: Mon Nov 19, 2007 3:55 pm UTC
Location: There may be lovelier lovelies...

Re: Problems in medical research:Tracking outcome switching

Postby Izawwlgood » Thu Feb 18, 2016 12:35 am UTC

mfb wrote:The fact that it happens frequently in some scientific disciplines doesn't make it better. It does not happen in particle physics, for example.
Probably because particle physics requires multi-million dollar machines be built, that takes potentially a decade and involves hundreds of researchers. What that machine does is published, and those data are far less fuzzy.

Very different in the life sciences.

mfb wrote:"Use this drug, we have 2 studies that show its effect (p<0.05)" and 50 studies that found no effect. Also, it will stop you using a drug that actually does work.
Not at all?
Yeah, I think your hyperbolic point isn't really how drug development works. Companies aren't interested in selling drugs that don't do anything, because getting it to the point of approval and marketing and convincing doctors to prescribe it is an expensive long term project. No one is going to push through something that doesn't actually do anything, knowing full well that another better drug can come in and swoop it.
... with gigantic melancholies and gigantic mirth, to tread the jeweled thrones of the Earth under his sandalled feet.

User avatar
DaBigCheez
Posts: 838
Joined: Tue Jan 04, 2011 8:03 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby DaBigCheez » Thu Feb 18, 2016 1:48 am UTC

mfb wrote:
Izawwlgood wrote:'Cheap in comparison' does not remotely mean 'cheap'.
Yes, but it is like throwing a new car away because you would have to buy a steering wheel to use it. If you are fine with "just making it public" instead of a peer-reviewed publication, it is even cheaper (just the costs of writing the results down).

I feel like this analogy is flawed by the fact that, in the analogy, there is a large benefit waiting for you (the car) with a small amount of additional investment (the steering wheel), whereas the problem under discussion is that publishing a null result provides no direct benefit to those that have done the study, and substantial benefit to their competitors, while still incurring that small additional cost. If publishing a null study directly translated to more revenue, we wouldn't even be having this discussion, I suspect.
existential_elevator wrote:It's like a jigsaw puzzle of Hitler pissing on Mother Theresa. No individual piece is offensive, but together...

If you think hot women have it easy because everyone wants to have sex at them, you're both wrong and also the reason you're wrong.

mfb
Posts: 950
Joined: Thu Jan 08, 2009 7:48 pm UTC

Re: Problems in medical research:Tracking outcome switching

Postby mfb » Sat Feb 20, 2016 2:46 pm UTC

Izawwlgood wrote:
mfb wrote:The fact that it happens frequently in some scientific disciplines doesn't make it better. It does not happen in particle physics, for example.
Probably because particle physics requires multi-million dollar machines be built, that takes potentially a decade and involves hundreds of researchers. What that machine does is published, and those data are far less fuzzy.

Very different in the life sciences.
I don't see how that difference would matter.
There are thousands of studies you could do with the detectors that are not done, because (a) you know other experiments are way better doing that or (b) there is not enough manpower for it. But if the study is done, the result is made public. Why not? Who has a legitimate interest in hiding scientific results?

Izawwlgood wrote:
mfb wrote:"Use this drug, we have 2 studies that show its effect (p<0.05)" and 50 studies that found no effect. Also, it will stop you using a drug that actually does work.
Not at all?
Yeah, I think your hyperbolic point isn't really how drug development works. Companies aren't interested in selling drugs that don't do anything, because getting it to the point of approval and marketing and convincing doctors to prescribe it is an expensive long term project. No one is going to push through something that doesn't actually do anything, knowing full well that another better drug can come in and swoop it.
If a drug went through that process already, then I am sure companies are interested in keeping selling it.
But even if the companies know about its uselessness in advance... or do you really think companies selling homeopathic stuff believe in that nonsense?


DaBigCheez wrote:whereas the problem under discussion is that publishing a null result provides no direct benefit to those that have done the study
Right, and that is a problem.

HungryHobo
Posts: 1708
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby HungryHobo » Sat Feb 20, 2016 11:46 pm UTC

Izawwlgood wrote:No one is going to push through something that doesn't actually do anything, knowing full well that another better drug can come in and swoop it.


Drugs have been withdrawn worldwide due to simple lack of efficacy.
It's what you get when drugs get pushed through on the back of crap trials.

Once you've spent hundreds of millions getting to stage 3 trials there's a strong incentive to corrupt a few naive researchers and skew a few results if it means making back the cost's to that point.

The marketing goons don't give a flying fuck if the drug works or not, all they care about is whether they can sell it. If they can find a handful of corrupt of simply incompetent scientists they can make it look like it works as long as nobody bothers publishing boring insignificant results.

Failing to report trial results absolutely should be considered research misconduct, it's simply common research misconduct that's rarely noticed or punished.
In human trials it's straight up unethical since the trial participants were put at risk and if the data generated is never published then you've just experimented on humans for no purpose.

If you collected data from 20 groups and hid the data from the 19 which didn't support your hypothesis there would be no question that you were committing research misconduct and effectively falsifying data.
If you collected data from 20 groups, split it into 20 papers then hid the data from the 19 which didn't support your hypothesis then it's little different, it's just slightly harder to prove intentional misconduct and looks very similar to simple laziness and being bad at your job.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.

aph
Posts: 296
Joined: Tue Nov 12, 2013 7:48 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby aph » Sat Feb 20, 2016 11:54 pm UTC

HungryHobo wrote:Drugs have been withdrawn worldwide due to simple lack of efficacy.
It's what you get when drugs get pushed through on the back of crap trials.

Once you've spent hundreds of millions getting to stage 3 trials there's a strong incentive to corrupt a few naive researchers and skew a few results if it means making back the cost's to that point.

The marketing goons don't give a flying fuck if the drug works or not, all they care about is whether they can sell it. If they can find a handful of corrupt of simply incompetent scientists they can make it look like it works as long as nobody bothers publishing boring insignificant results.

Failing to report trial results absolutely is research misconduct, it's simply common research misconduct that's rarely punished.

If you collected data from 20 groups and hid the data from the 19 which didn't support your hypothesis there would be no question that you were committing research misconduct and effectively falsifying data.
If you collected data from 20 groups, split it into 20 papers then his the data from the 19 which didn't support your hypothesis then it's little different, it's just slightly harder to prove intentional misconduct and looks very similar to simple laziness.

Damn. So any drug will just get tested on patients, no matter the millions in trials. Maybe that is unavoidable.

qetzal
Posts: 862
Joined: Thu May 01, 2008 12:54 pm UTC

Re: Problems in medical research:Tracking outcome switching

Postby qetzal » Sun Feb 21, 2016 12:01 am UTC

HungryHobo wrote:Once you've spent hundreds of millions getting to stage 3 trials there's a strong incentive to corrupt a few naive researchers and skew a few results if it means making back the cost's to that point. The marketing goons don't give a flying fuck if the drug works or not, all they care about is whether they can sell it. If they can find a handful of corrupt of simply incompetent scientists they can make it look like it works as long as nobody bothers publishing boring insignificant results.


Yeah, that doesn't really work in countries where you have to convince a regulatory (FDA, EMA) to approve your drug. And if you're not in one of those countries, then you're not bothering to spend hundreds of millions on Phase 3 trials.

HungryHobo
Posts: 1708
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby HungryHobo » Sun Feb 21, 2016 12:27 am UTC

qetzal wrote:Yeah, that doesn't really work in countries where you have to convince a regulatory (FDA, EMA) to approve your drug. And if you're not in one of those countries, then you're not bothering to spend hundreds of millions on Phase 3 trials.


Except that in the real world crap drugs still get through.

Tamiflu is a pretty good example.
Governments spent billions stockpiling Tamiflu based on published trials at the time. (yes, including, indeed primarily in countries which had bodies like the FDA and EMA)

Then it turned out that the drug company had run far far more trials than they had publicly disclosed, when they were forced to show the results after years of fighting it turned out that the drug was basically useless for reducing deaths and serious complications from flu (the whole point) when taking into account the results of those trials.

http://community.cochrane.org/features/ ... ce-picture

“We now have the most robust, comprehensive review on neuraminidase inhibitors that exists. Initially thought to reduce hospitalisations and serious complications from influenza, the review highlights that [NIs are] not proven to do this, and it also seems to lead to harmful effects that were not fully reported in the original publications. This shows the importance of ensuring that trial data are transparent and accessible.”


You don't get good science just by chanting "FDA FDA FDA" in front of a mirror in the dark.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.

qetzal
Posts: 862
Joined: Thu May 01, 2008 12:54 pm UTC

Re: Problems in medical research:Tracking outcome switching

Postby qetzal » Sun Feb 21, 2016 1:16 am UTC

Yes, crap drugs to get through. I don't dispute that. I dispute that crap drugs frequently get through because drug companies can just hire a few corrupt, naïve, or incompetent researchers.

lorb
Posts: 405
Joined: Wed Nov 10, 2010 10:34 am UTC
Location: Austria

Re: Problems in medical research:Tracking outcome switching

Postby lorb » Sun Feb 21, 2016 2:53 am UTC

In the current regime that is actually impossible to know. Nobody really knows how much unpublished data there is that would prove that drugs that are currently in use lack efficacy.
Please be gracious in judging my english. (I am not a native speaker/writer.)
http://decodedarfur.org/

qetzal
Posts: 862
Joined: Thu May 01, 2008 12:54 pm UTC

Re: Problems in medical research:Tracking outcome switching

Postby qetzal » Sun Feb 21, 2016 4:11 am UTC

Well first of all, hiding negative date is not the same as hiring corrupt or incompetent researchers to generate fraudulent data, which was HungryHobo's claim. Second, drug developers can and do manipulate which data they publish, but it's much harder for them to conceal negative data from the regulators. That's even more true now that regulators require up-front registration of any trials that a developer hopes to use to gain approval. You can't start 20 trials, hoping to get approval based only on the 1 that randomly meets p < 0.05, because you'd have to register all of them. And the regulators will ask you for results on them all.

In fact, HungryHobo's link about Tamiflu makes that point pretty clearly:

From the Plain Language Summary (see http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD008965.pub4/abstract;jsessionid=D0952B85723A29FF11B3087DB5F71938.f03t02):

In previous versions of this review we identified unresolved discrepancies in the data presented in published trial reports and substantial publication bias. As a consequence, we elected not to use data from journal articles but included the documents generated during licensing processes. We have accessed such data from the UK, USA, European Medicines Agency (EMA), Japanese regulators and clinical study reports from the manufacturers (after a protracted media campaign). This has enabled us to verify information from the randomised, placebo-controlled trials on adults and children with confirmed or suspected exposure to naturally occurring influenza.

Based on our assessments of the regulatory documents (in excess of 160,000 pages), we came to the conclusion that there were substantial problems with the design, conduct, reporting and availability of information from many of the trials.

[snip]

The lack of good evidence demonstrating an effect on complications agrees with the conservative conclusions on both drugs drawn by the US Food and Drug Administration (FDA). The FDA only allowed claims of effectiveness of both drugs for the prevention and treatment of symptoms of influenza and not for other effects (including the interruption of person-to-person spread of the influenza virus or prevention of pneumonia). The FDA described the overall performance of both drugs as 'modest'.


The point being that even if the neuraminidase inhibitor developers were trying to conceal much of that negative data from the general public & from most MDs & researchers, FDA still had access to all of it. That was where Cochran got the data to draw their conclusions in the first case. And FDA did not allow them to make any claims about preventing complications, precisely because of all that negative data that they did review.

Of course, that doesn't mean the system actually works well. It would be far better if the results of those trials were made available to the MDs, researchers, and the public, not just to the FDA. Absent that, drug companies (& their employees) will absolutely try to get their drugs used as much as possible, even under circumstances that may not be appropriate.

In fact, there are efforts to try to 'force' publication of all clinical trials, whether they're run by drug companies or by academics. One interesting irony is that the drug companies currently seem to be better at this than academics. I don't have the citation to hand right now, but I believe I read about it over at Derek Lowe's In The Pipeline blog, if anyone wants to try to find it.

HungryHobo
Posts: 1708
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby HungryHobo » Sun Feb 21, 2016 8:26 am UTC

In theory it should now be a solved problem. In theory.

First, while the rules say that trials should be pre-registered in reality trials often only get registered shortly before publication, long after the trial began when they may already have had a chance to p-hack.
Second, while officially there are penalties for failing to give the results of human trials that have already begun, they've never ever been applied to anyone and that isn't for lack of violations.
Third, even for trials which were pre-registered in a huge proportion of cases researchers simply ignore what they preregistered and silently switch the outcomes to get a significant result.

The third case is where compare-trials.org comes in.

The situation is far better than it was even a few years ago. The fact that we even know about the third case is, in a way, far better than the alternative but there's still a long way to go.

Fourth, there are still often gaping holes in the regulatory frameworks. Up until recently in the UK drug companies were only required to submit all trials they'd done to the regulator for the group for which they were seeking market approval.
Most drugs are used off label regularly unless there are contra-indications in the literature since it's impractical to do a thousand trials for every subset of the population. If, say, an anti-cancer drug has been proven to be effective in experiments on adults and there's no evidence that it's harmful to children it would be unethical to withhold it from dying children. Most of the time this works fine.

So a company sought approval for a drug, it worked quite well according to the trials they submitted to the regulator and it was approved. There was no evidence available available to doctors that it was dangerous to children. Years later it turned out that the company had in fact done safety trials with children and found that it was dangerous to children. And they hid the data. They didn't publish it, they didn't submit it to the regulator and then they sold the drug in such a way that they knew that it would be prescribed to children when they knew that their drug was dangerous to children. Anyone here believe that the researchers involved were, ethically, in the clear for keeping the results of those trials unpublished? Keep in mind that those same researchers would have then watched the drug go to market and be made available to children.

That specific loophole I believe was closed in the UK because of that event but it's a good example of how simply shouting "regulators" doesn't solve the problem.
It only came to light because a court case with the company brought up some documents showing that the trials existed into the public sphere.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.

User avatar
Izawwlgood
WINNING
Posts: 18686
Joined: Mon Nov 19, 2007 3:55 pm UTC
Location: There may be lovelier lovelies...

Re: Problems in medical research:Tracking outcome switching

Postby Izawwlgood » Sun Feb 21, 2016 3:03 pm UTC

I feel like this bears repeating, because I'm not sure the point was understood -

The NIH requires that all trials that involve human subjects be reported to clinicaltrials.gov. Failure to do so results in fines and potentially a loss of the ability to conduct further trials. If this results in people taking trials abroad, that's one thing, but this is the regulatory step that has been taken to prevent trial spamming.
... with gigantic melancholies and gigantic mirth, to tread the jeweled thrones of the Earth under his sandalled feet.

HungryHobo
Posts: 1708
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby HungryHobo » Sun Feb 21, 2016 4:07 pm UTC

Izawwlgood wrote:I feel like this bears repeating, because I'm not sure the point was understood -

The NIH requires that all trials that involve human subjects be reported to clinicaltrials.gov. Failure to do so results in fines and potentially a loss of the ability to conduct further trials. If this results in people taking trials abroad, that's one thing, but this is the regulatory step that has been taken to prevent trial spamming.


yes, I already covered that under "Second"

Under FDA regulations drug and device clinical trials, done in the US, must report their results within 1 year of completion on the website ClinicalTrials.gov
If you don’t report your results, you're supposed to be fined for every day that you don’t.
However, to date this fine has never been imposed. Ever.
About half of all trials violate this law and nobody has ever been fined a single cent.

A law that is never enforced is just ink and paper.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.

User avatar
Izawwlgood
WINNING
Posts: 18686
Joined: Mon Nov 19, 2007 3:55 pm UTC
Location: There may be lovelier lovelies...

Re: Problems in medical research:Tracking outcome switching

Postby Izawwlgood » Sun Feb 21, 2016 5:40 pm UTC

My understanding is that the goal is to get trials reported, not to fine companies/researchers. As a result, when a trial fails to report, the follow up is a threat of a fine to get them to. But yeah, it's a slow process and clinical-trials.gov is still playing catch up.

Worth pointing out though that private companies seem to be better than government funded researchers.
... with gigantic melancholies and gigantic mirth, to tread the jeweled thrones of the Earth under his sandalled feet.

HungryHobo
Posts: 1708
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby HungryHobo » Sun Feb 21, 2016 6:05 pm UTC

Izawwlgood wrote:My understanding is that the goal is to get trials reported, not to fine companies/researchers. As a result, when a trial fails to report, the follow up is a threat of a fine to get them to. But yeah, it's a slow process and clinical-trials.gov is still playing catch up.

Worth pointing out though that private companies seem to be better than government funded researchers.


People quickly learn to recognise paper tigers.

Comparing like with like or just in terms of raw percentages?

ie, comparing 10 million dollar trials with 10 million dollar trials or comparing 10 million dollar trials with Phd student Bob and whatever pocket change his supervisor can scrape together?
I wouldn't be remotely surprised if lots of small trials run by a Phd student with a tiny budget fall apart and/or don't get properly reported.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.

User avatar
Izawwlgood
WINNING
Posts: 18686
Joined: Mon Nov 19, 2007 3:55 pm UTC
Location: There may be lovelier lovelies...

Re: Problems in medical research:Tracking outcome switching

Postby Izawwlgood » Sun Feb 21, 2016 7:01 pm UTC

HungryHobo wrote:I wouldn't be remotely surprised if lots of small trials run by a Phd student with a tiny budget fall apart and/or don't get properly reported.
Yeah, I'm pretty sure at this point you don't understand how clinical trials are run if you think this is a thing, or if you think some low budget project is the sort of thing we're talking about.

But that aside, the point isn't that these fines are effectively hollow threats, but that the process of requiring reporting is only ~7 years old. At this point, the effort is still in its infancy. In 2007 there were about 26,000 investigators registering for clinical trials. This is no small effort to get everyone to report things, especially given how long many trials will run for.
... with gigantic melancholies and gigantic mirth, to tread the jeweled thrones of the Earth under his sandalled feet.

ijuin
Posts: 1152
Joined: Fri Jan 09, 2009 6:02 pm UTC

Re: Problems in medical research:Tracking outcome switching

Postby ijuin » Sun Feb 21, 2016 7:47 pm UTC

Yes--Stage III clinical trials generally involve hundreds of test subjects in order to get a statistically significant sample. It is NOT simply a grad student running around taking monthly surveys of a group of people--it is hundreds of people taking a drug or using a medical device for several months to a year.

HungryHobo
Posts: 1708
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby HungryHobo » Sun Feb 21, 2016 9:40 pm UTC

Izawwlgood wrote:Yeah, I'm pretty sure at this point you don't understand how clinical trials are run if you think this is a thing, or if you think some low budget project is the sort of thing we're talking about.


We've got small projects done largely by Phd students out of my department testing interventions and I'm quite familiar with larger scale trials.
If you believe such small trials don't exist inform yourself better.

It's a simple question.
That kind of claim without citation could lump the 2 together.
Presumably the 50% claim came from somewhere, some research or report that lists it's inclusion criteria?

The discussion is not limited to stage 3 trials.
Hiding data in earlier stages of research is also a problem and researchers are bound by the same ethical obligations to disclose results from trials on human subjects.

Spoiler:
Izawwlgood: There's no need for the typical snippy, unpleasant, insulting bullshit which appears to be the only thing of substance you ever seem to bring to topics. Stop trying to start a fucking flamewar with your typical trollish shit, I'd prefer to have a discussion addressing actual claims and facts rather than your half-informed arrogant out of date opinions mixed with insults. I posted a civil fucking question and the only way you could respond was by being insulting? Go fuck yourself.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.

User avatar
Izawwlgood
WINNING
Posts: 18686
Joined: Mon Nov 19, 2007 3:55 pm UTC
Location: There may be lovelier lovelies...

Re: Problems in medical research:Tracking outcome switching

Postby Izawwlgood » Mon Feb 22, 2016 4:36 am UTC

I don't think smaller trials are particularly relevant to this discussion. Small independent research projects by a couple of grad students isn't how drug safety is tested, and remember, the issue here is tracking outcome switching and how it pertains to misrepresentation of data.
... with gigantic melancholies and gigantic mirth, to tread the jeweled thrones of the Earth under his sandalled feet.

User avatar
gmalivuk
GNU Terry Pratchett
Posts: 26836
Joined: Wed Feb 28, 2007 6:02 pm UTC
Location: Here and There
Contact:

Re: Problems in medical research:Tracking outcome switching

Postby gmalivuk » Mon Feb 22, 2016 2:09 pm UTC

HungryHobo: The thing in this thread that looks most like trying to start a flame war with trollish bullshit is that spoiler from you. Cut it out or leave the thread. (No I don't care that you started it.)
Unless stated otherwise, I do not care whether a statement, by itself, constitutes a persuasive political argument. I care whether it's true.
---
If this post has math that doesn't work for you, use TeX the World for Firefox or Chrome

(he/him/his)

p1t1o
Posts: 960
Joined: Wed Nov 10, 2010 4:32 pm UTC
Location: London, UK

Re: Problems in medical research:Tracking outcome switching

Postby p1t1o » Mon Feb 22, 2016 3:30 pm UTC

HungryHobo wrote:So a company sought approval for a drug, it worked quite well according to the trials they submitted to the regulator and it was approved. There was no evidence available available to doctors that it was dangerous to children. Years later it turned out that the company had in fact done safety trials with children and found that it was dangerous to children. And they hid the data. They didn't publish it, they didn't submit it to the regulator and then they sold the drug in such a way that they knew that it would be prescribed to children when they knew that their drug was dangerous to children. Anyone here believe that the researchers involved were, ethically, in the clear for keeping the results of those trials unpublished? Keep in mind that those same researchers would have then watched the drug go to market and be made available to children.


This may or may not be relevant, but you seem to be under the impression that the people in white coats making notes in the lab are physically the same people that put together marketing strategies.
A cluster-%&@# on the scale described can hardly have been so simple as to describe in a single paragraph anyway, what was the name of the company?

HungryHobo
Posts: 1708
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: Problems in medical research:Tracking outcome switching

Postby HungryHobo » Thu Feb 25, 2016 3:17 pm UTC

p1t1o wrote: what was the name of the company?


GSK. They were fined 3 billion in the US for hiding data, bribing doctors, and breaking marketing laws related to that drug and others. Specifically one of the issues that came up in the case was marketing to convince doctors to prescribe to children. (after they had done the trials internally that showed it to be dangerous to children)
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.

p1t1o
Posts: 960
Joined: Wed Nov 10, 2010 4:32 pm UTC
Location: London, UK

Re: Problems in medical research:Tracking outcome switching

Postby p1t1o » Thu Feb 25, 2016 4:21 pm UTC

Oh GSK, that does make sense, as far as the term "Big Pharma" goes, they basically invented it.

I don't like the term myself, GSK are not the only pharma company and most have much less chequered history. Tarring a whole group with the same brush is a pet peeve of mine.

Its still not necessarily the case that the same person trying to get people to sell to children had the data that said it was harmful to children, conversely, the people WITH the data were not necessarily aware that the marketeers were targetting children.

I'm not trying to say the whole thing was made up, just that large companies don't work perfectly, and to assume that the marketting and research departments were working in perfect synch is a huge assumption.

What they were probably guilty of was negligence rather than literally trying to poison children for profit.


There is a story in the news at the moment, Johnson & Johnson have been sued successfully (awaiting appeal) for $72million because a lady died of ovarian cancer whilst using copious amounts of Johnsons baby powder (talc).
The evidence for talc carcinogenicity is pretty inconclusive and no regulatory body has ever classified it as such. There is evidence that it can cause cancer if used on the (female) genitalia, but there is just as much evidence that it doesn't.
But the court had a memo from a single medical professional where he compares talcs use on female genitalia to smoking cigarettes.

Keep in mind, J&J are not the only people who sell talc, and talc is an extremely common household product that has been in use for decades.

Now, are they supposed to pull their talc from shelves or put "Danger: carcinogenic" on the bottle because of one unsubstantiated memo? I work in regulatory and I used to work for J&J and I can say that I certainly wouldn't have, though I probably would have asked the doctor to provide documentation to back up his "memo", if it came to me.

This example was just to illustrate what a bas*^rd it is regulating these things.

(By all reports the appeal is predicted to be successful and they are unlikely to have to pay.)

lorb
Posts: 405
Joined: Wed Nov 10, 2010 10:34 am UTC
Location: Austria

Re: Problems in medical research:Tracking outcome switching

Postby lorb » Thu Feb 25, 2016 4:50 pm UTC

p1t1o wrote:I don't like the term myself, GSK are not the only pharma company and most have much less chequered history. Tarring a whole group with the same brush is a pet peeve of mine.


Still there is a reason why this exists: Wikipedias list of largest pharmaceutical settlements

I am not saying all pharmaceutical companies are evil, but there are definitely some things that are systematically going wrong in their business. Leading back to the topic: researches feeling pressed to come up with significant results is one of those things.
Please be gracious in judging my english. (I am not a native speaker/writer.)
http://decodedarfur.org/


Return to “Science”

Who is online

Users browsing this forum: No registered users and 9 guests