About this Author
DBL%20Hendrix%20small.png College chemistry, 1983

Derek Lowe The 2002 Model

Dbl%20new%20portrait%20B%26W.png After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. To contact Derek email him directly: Twitter: Dereklowe

Chemistry and Drug Data: Drugbank
Chempedia Lab
Synthetic Pages
Organic Chemistry Portal
Not Voodoo

Chemistry and Pharma Blogs:
Org Prep Daily
The Haystack
A New Merck, Reviewed
Liberal Arts Chemistry
Electron Pusher
All Things Metathesis
C&E News Blogs
Chemiotics II
Chemical Space
Noel O'Blog
In Vivo Blog
Terra Sigilatta
BBSRC/Douglas Kell
Realizations in Biostatistics
ChemSpider Blog
Organic Chem - Education & Industry
Pharma Strategy Blog
No Name No Slogan
Practical Fragments
The Curious Wavefunction
Natural Product Man
Fragment Literature
Chemistry World Blog
Synthetic Nature
Chemistry Blog
Synthesizing Ideas
Eye on FDA
Chemical Forums
Symyx Blog
Sceptical Chymist
Lamentations on Chemistry
Computational Organic Chemistry
Mining Drugs
Henry Rzepa

Science Blogs and News:
Bad Science
The Loom
Uncertain Principles
Fierce Biotech
Blogs for Industry
Omics! Omics!
Young Female Scientist
Notional Slurry
Nobel Intent
SciTech Daily
Science Blog
Gene Expression (I)
Gene Expression (II)
Adventures in Ethics and Science
Transterrestrial Musings
Slashdot Science
Cosmic Variance
Biology News Net

Medical Blogs
DB's Medical Rants
Science-Based Medicine
Respectful Insolence
Diabetes Mine

Economics and Business
Marginal Revolution
The Volokh Conspiracy
Knowledge Problem

Politics / Current Events
Virginia Postrel
Belmont Club
Mickey Kaus

Belles Lettres
Uncouth Reflections
Arts and Letters Daily
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline

In the Pipeline

« Wipe Down | Main | The Dull Edge of Nanotech »

June 15, 2004

The Journals Fight Back

Email This Entry

Posted by Derek

So it turns out that the major medical journals have their own plan for bringing on a clinical trial database: they're going to require companies to register trials before they'll allow publication of their results. I was taken aback at not having heard anything about this idea, until I saw that no one else in the drug industry seems to have, either.

I don't really have a problem with this at all. For one thing, it's better than having the state sue you into doing something - this is a good old free-market fight. Most of the major medical journals need revenue from pharmaceutical advertising, and the companies need the prestige of publishing in them. Come, then, let us reason together.

And the first step here, merely registering the fact of a trial, will sidestep some of the issues I brought up the other day with how to report the final data. I know that there will be pressure to include that data as well, and if we can find a way to deal with those reporting issues, we should. But even a registry of trials would show that something had been tried, naturally leading to questions about how things came out. (That's important for the medical editors' side of this dispute, because the studies that companies don't want to talk about aren't going to be submitted for publication, anyway - the journals have no other leverage at that point.)

Now, one way around this would be for companies to forsake publication in the journals involved (a tough thing to do, mind you) and just present the data with a big splash at a prestigious meeting or two. If you see more professional societies joining this trial-registry movement, especially ones that don't publish their own journals but still sponsor large meetings, then I think the outcome will have become clear.

I think, though, that people have some odd ideas about how clinical trials work and how many of them there are. Consider columnist Michelle Malkin, who wrote about this story today:

From Statistics 101 we know that if a product is as effective as a placebo, 1 in 20 trials will produce a statistically significant finding due to random chance. Since companies run dozens of trials on each major compound, it is not too hard to produce at least one positive, statistically significant finding suitable for publication. The rest are buried in the "circular file." This is great marketing but it is not science.

Um, we don't actually run "dozens" of trials on every major compound. We don't have enough money to do that, as hard as that may be to believe, and in many cases there just aren't enough patients to go around. So we just don't get to play with the statistics in this way. It would be irresponsible, she's right about that, but we don't do it.

And that argument would only hold if all 20 trials were run the exact same way (Statistics 101, you know.) Twenty different trials, each run a different way on different patient groups, can produce results all over the map. Trying to do metastatistics over the whole group is not a job you want; it's often not even possible. And besides, even if they were all the same, the level of statistical significance that Malkin's talking about (1 in 20 by random chance) isn't very high at all. A clinical trial has to be a lot more significant than that to convince anyone at either the FDA or the company itself.

Comments (5) + TrackBacks (0) | Category: Clinical Trials


1. qetzal on June 15, 2004 10:44 PM writes...

Another problem with Malkin's argument. If she's thinking about unapproved drugs, it's nonsense. Cause even if you decide to only publish the 1 in 20 that was positive by chance, you'd still have to show the FDA all 20 in your NDA. And of course, FDA isn't going to approve a drug that failed 19 trials out of 20. So, you run 20 trials just to get a publication? If big pharma's goal was to publish papers, they'd be universities.

On the other hand, if the drug is already approved for a given indication, then the idea is, um, nonsense. In theory at least, if it's approved, it's already proven to work in at least one indication. No value in running 20 more trials for that indication.

Aha, you say! What if it's approved for one indication, but the 20 trials are for a different one? Then see case one, above.

Or, what if the company runs 20 trials in 20 different indications, hoping one will hit? Well, it really isn't that common for a drug to be approved for a given use based on only one trial's evidence that it works. If I have it right, FDA typically likes to see either *two* successful trials for a given indication, or one clearly successful trial *plus* additional supportive data indicating the drug works.

So no matter how you cut it, even if a company wanted to do that, and was willing to spend the money, all they'd get is a 3rd rate pub in Journal of Uninteresting Clinical Results. They'd also go out of business quickly, wasting their money like that.

Permalink to Comment

2. Michelle Malkin on June 16, 2004 3:36 AM writes...

I cannot confirm that companies do dozens of trials on every major compound--I'll admit that was speculation on my part, and I'll take Derek's word for it that I was wrong about this--but I know for a fact that at least one company has run dozens of trials on one of its products (an FDA-approved antibiotic). My source tells me that most of these trial results will never be published in a journal.

Trials need not be identical to achieve statistical significance by chance. If the product has no effect, each trial would have a 5% chance of producing a statistically significant result, no matter what the design.

As I noted in my original post, one of the reasons to support the journals' proposal is that companies sometimes change the endpoint after they see the results (e.g., Pfizer-funded scientists used a 6 month endpoint when the 12 month endpoint in a Celebrex trial didn't show positive results). In effect, companies are running several sub-trials within each trial, picking and choosing what to publish based on the results. This is not sound science.

If the FDA rejects the findings of a study, it may have little effect if a reputable journal has already published positive results. This is what happened in the case of Celebrex referenced above. Even if the results aren't submitted for FDA approval, they can still be helpful to the company if published. (Yes, I know it is illegal for a company to market a drug for an FDA-unapproved use, but it does happen.)

Permalink to Comment

3. Michelle Malkin on June 16, 2004 5:46 AM writes...

Correction: should have said "more than a dozen trials" not "dozens of trials."

Permalink to Comment

4. Derek Lowe on June 16, 2004 10:26 AM writes...

Antibiotics are a bit of a special case, because there are so many potential different patient populations. The FDA wants separate studies if you want labeling for, say, gram-negative respiratory infections, or such-and-such-resistant urinary tract infections. Thus, lots of trials. Oncology is like this, too, but there the patient population is so desperate that everyone tries everything for everything, without waiting for the data at all.

You're right to be wary of subgroup analysis, because in a large enough trial you can always find (by pure chance) some subgroup that seemed to show significance. Run it again, and that group disappears, only to be replaced by another false positive. (There's your statistical argument in a different form, one I can agree with more.)

But this is something the FDA constantly has its eye out for - they know that trick as well as anyone can. And as Qetzal points out above, you have to tell the FDA about all your studies for a given indication, positive or not, when you go to them for new labeling approval.

And yep, off-label marketing goes on all the time, and it can even take off by itself without much help from the company (as in that oncology exampble above.) In those cases, the company won't have even run a trial for the indication that doctors are writing scripts for. But that's a topic for another day and another post. Several, actually.

Permalink to Comment

5. kevin white on June 16, 2004 12:29 PM writes...

If pharmas were constantly shopping for statistical signifigance wouldn't we be seeing a large number of drugs declared as useless years after they were introduced? It would also be easy for a pharma to eliminate a competitor by sponsoring studies that show zero efficacy. We're not really seeing either one of those things happen.

This of course assumes that there isn't some global conspiracy by pharmas to not poach on each other's territory. Since pharmas do try to introduce competing products all the time it doesn't really seem like a conspiracy is occuring. Even OPEC keeps better control over it's members.

Permalink to Comment


Email this entry to:

Your email address:

Message (optional):

The Last Post
The GSK Layoffs Continue, By Proxy
The Move is Nigh
Another Alzheimer's IPO
Cutbacks at C&E News
Sanofi Pays to Get Back Into Oncology
An Irresponsible Statement About Curing Cancer
Oliver Sacks on Turning Back to Chemistry