FREE NEJM E-TOC    HOME   |   SUBSCRIBE   |   CURRENT ISSUE   |   PAST ISSUES   |   COLLECTIONS   |   Search Term  Advanced Search

Sign in | Get NEJM's E-Mail Table of Contents — Free | Subscribe

Previous Volume 358:252-260 January 17, 2008 Number 3 Next

Abstract PDF PDA Full Text PowerPoint Slide Set Supplementary Material Letters Add to Personal Archive Add to Citation Manager Notify a Friend E-mail When Cited E-mail When Letters Appear PubMed Citation

ABSTRACT

Background Evidence-based medicine is valuable to the extent that the evidence base is complete and unbiased. Selective publication of clinical trials — and the outcomes within those trials — can lead to unrealistic estimates of drug effectiveness and alter the apparent risk–benefit ratio.

Methods We obtained reviews from the Food and Drug Administration (FDA) for studies of 12 antidepressant agents involving 12,564 patients. We conducted a systematic literature search to identify matching publications. For trials that were reported in the literature, we compared the published outcomes with the FDA outcomes. We also compared the effect size derived from the published reports with the effect size derived from the entire FDA data set.

Results Among 74 FDA-registered studies, 31%, accounting for 3449 study participants, were not published. Whether and how the studies were published were associated with the study outcome. A total of 37 studies viewed by the FDA as having positive results were published; 1 study viewed as positive was not published. Studies viewed by the FDA as having negative or questionable results were, with 3 exceptions, either not published (22 studies) or published in a way that, in our opinion, conveyed a positive outcome (11 studies). According to the published literature, it appeared that 94% of the trials conducted were positive. By contrast, the FDA analysis showed that 51% were positive. Separate meta-analyses of the FDA and journal data sets showed that the increase in effect size ranged from 11 to 69% for individual drugs and was 32% overall.

Conclusions We cannot determine whether the bias observed resulted from a failure to submit manuscripts on the part of authors and sponsors, from decisions by journal editors and reviewers not to publish, or both. Selective reporting of clinical trial results may have adverse consequences for researchers, study participants, health care professionals, and patients.

Attempts to study selective publication are complicated by the unavailability of data from unpublished trials. Researchers have found evidence for selective publication by comparing the results of published trials with information from surveys of authors,⁵ registries,⁶ institutional review boards,^7,8 and funding agencies,^9,10 and even with published methods.¹¹ Numerous tests are available to detect selective-reporting bias, but none are known to be capable of detecting or ruling out bias reliably.^{12,13,14,15,16}

In the United States, the Food and Drug Administration (FDA) operates a registry and a results database.¹⁷ Drug companies must register with the FDA all trials they intend to use in support of an application for marketing approval or a change in labeling. The FDA uses this information to create a table of all studies.¹⁸ The study protocols in the database must prospectively identify the exact methods that will be used to collect and analyze data. Afterward, in their marketing application, sponsors must report the results obtained using the prespecified methods. These submissions include raw data, which FDA statisticians use in corroborative analyses. This system prevents selective post hoc reporting of favorable trial results and outcomes within those trials.

How accurately does the published literature convey data on drug efficacy to the medical community? To address this question, we compared drug efficacy inferred from the published literature with drug efficacy according to FDA reviews.

Methods

Data from FDA Reviews

We identified the phase 2 and 3 clinical-trial programs for 12 antidepressant agents approved by the FDA between 1987 and 2004 (median, August 1996), involving 12,564 adult patients. For the eight older antidepressants, we obtained hard copies of statistical and medical reviews from colleagues who had procured them through the Freedom of Information Act.¹⁹ Reviews for the four newer antidepressants were available on the FDA Web site.^17,20 This study was approved by the Research and Development Committee of the Portland Veterans Affairs Medical Center; because of its nature, informed consent from individual patients was not required.

From the FDA reviews of submitted clinical trials, we extracted efficacy data on all randomized, double-blind, placebo-controlled studies of drugs for the short-term treatment of depression. We included data pertaining only to dosages later approved as safe and effective; data pertaining to unapproved dosages were excluded.

We extracted the FDA's regulatory decisions — that is, whether, for purposes of approval, the studies were judged to be positive or negative with respect to the prespecified primary outcomes (or primary end points).²¹ We classified as questionable those studies that the FDA judged to be neither positive nor clearly negative — that is, studies that did not have significant findings on the primary outcome but did have significant findings on several secondary outcomes. Failed studies²² were also classified as questionable (for more information, see the Methods section of the Supplementary Appendix, available with the full text of this article at www.nejm.org). For fixed-dose studies (studies in which patients are randomly assigned to receive one of two or more dose levels or placebo) with a mix of significant and nonsignificant results for different doses, we used the FDA's stated overall decisions on the studies. We used double data extraction and entry, as detailed in the Methods section of the Supplementary Appendix.

Data from Journal Articles

Our literature-search strategy consisted of the following steps: a search of articles in PubMed, a search of references listed in review articles, and a search of the Cochrane Central Register of Controlled Trials; contact by telephone or e-mail with the drug sponsor's medical-information department; and finally, contact by means of a certified letter sent to the sponsor's medical-information department, including a deadline for responding in writing to our query about whether the study results had been published. If these steps failed to reveal any publications, we concluded that the study results had not been published.

We identified the best match between the FDA-reviewed clinical trials and journal articles on the basis of the following information: drug name, dose groups, sample size, active comparator (if used), duration, and name of principal investigator. We sought published reports on individual studies; articles covering multiple studies were excluded. When the results of a trial were reported in two or more primary publications, we selected the first publication.

Few journal articles used the term "primary efficacy outcome" or a reasonable equivalent. Therefore, we identified the apparent primary efficacy outcome, or the result highlighted most prominently, as the drug–placebo comparison reported first in the text of the results section or in the table or figure first cited in the text. As with the FDA reviews, we used double data extraction and entry (see the Methods section of the Supplementary Appendix for details).

Statistical Analysis

We categorized the trials on the basis of the FDA regulatory decision, whether the trial results were published, and whether the apparent primary outcomes agreed or conflicted with the FDA decision. We calculated risk ratios with exact 95% confidence intervals and Pearson's chi-square analysis, using Stata software, version 9. We used a similar approach to examine the numbers of patients within the studies. Sample sizes were compared between published and unpublished studies with the use of the Wilcoxon rank-sum test.

For our major outcome indicator, we calculated the effect size for each trial using Hedges's g — that is, the difference between two means divided by their pooled standard deviation.²³ However, because means and standard deviations (or standard errors) were inconsistently reported in both the FDA reviews and the journal articles, we used the algebraically equivalent computational equation²⁴:

g = t x the square root of (1/n_drug + 1/n_placebo).

We calculated the t statistic²⁵ using the precise P value and the combined sample size as arguments in Microsoft Excel's TINV (inverse T) function, multiplying t by –1 when the study drug was inferior to the placebo. Hedges's correction for small sample size was applied to all g values.²⁶

Precise P values were not always available for the above calculation. Rather, P values were often indicated as being below or above a certain threshold — for example, P<0.05 or "not significant" (i.e., P>0.05). In these cases, we followed the procedure described in the Supplementary Appendix.

For each fixed-dose (multiple-dose) study, we computed a single study-level effect size weighted by the degrees of freedom for each dose group. On the basis of the study-level effect-size values for both fixed-dose and flexible-dose studies, we calculated weighted mean effect-size values for each drug and for all drugs combined, using a random-effects model with the method of DerSimonian and Laird²⁷ in Stata.²⁸

Within the published studies, we compared the effect-size values derived from the journal articles with the corresponding effect-size values derived from the FDA reviews. Next, within the FDA data set, we compared the effect-size values for the published studies with the effect-size values for the unpublished studies. Finally, we compared the journal-based effect-size values with those derived from the entire FDA data set — that is, both published and unpublished studies.

We made these comparisons at the level of studies and again at the level of the 12 drugs. Because the data were not normally distributed, we used the nonparametric rank-sum test for unpaired data and the signed-rank test for paired data. In these analyses, all the effect-size values were given equal weight.

Results

Study Outcome and Publication Status

Of the 74 FDA-registered studies in the analysis we could not find evidence of publication for 23 (31%) (Table 1). The difference between the sample sizes for the published studies (median, 153 patients) and the unpublished studies (median, 146 patients) was neither large nor significant (5% difference between medians; P=0.29 by the rank-sum test).

View this table: [in this window] [in a new window]   Table 1. Overall Publication Status of FDA-Registered Antidepressant Studies.

 
The data in Table 1 are displayed in terms of the study outcome in Figure 1A. The questions of whether the studies were published and, if so, how the results were reported were strongly related to their overall outcomes. The FDA deemed 38 of the 74 studies (51%) positive, and all but 1 of the 38 were published. The remaining 36 studies (49%) were deemed to be either negative (24 studies) or questionable (12). Of these 36 studies, 3 were published as not positive, whereas the remaining 33 either were not published (22 studies) or were published, in our opinion, as positive (11) and therefore conflicted with the FDA's conclusion. Overall, the studies that the FDA judged as positive were approximately 12 times as likely to be published in a way that agreed with the FDA analysis as were studies with nonpositive results according to the FDA (risk ratio, 11.7; 95% confidence interval [CI], 6.2 to 22.0; P<0.001). This association of publication status with study outcome remained significant when we excluded questionable studies and when we examined publication status without regard to whether the published conclusions and the FDA conclusions were in agreement (for details, see the Supplementary Appendix).

View larger version (20K): [in this window] [in a new window]   Figure 1. Effect of FDA Regulatory Decisions on Publication. Among the 74 studies reviewed by the FDA (Panel A), 38 were deemed to have positive results, 37 of which were published with positive results; the remaining study was not published. Among the studies deemed to have questionable or negative results by the FDA, there was a tendency toward nonpublication or publication with positive results, conflicting with the conclusion of the FDA. Among the 12,564 patients in all 74 studies (Panel B), data for patients who participated in studies deemed positive by the FDA were very likely to be published in a way that agreed with the FDA. In contrast, data for patients participating in studies deemed questionable or negative by the FDA tended either not to be published or to be published in a way that conflicted with the FDA's judgment.

 
Overall, 48 of the 51 published studies were reported to have positive results (94%; binomial 95% CI, 84 to 99). According to the FDA, 38 of the 74 registered studies had positive results (51%; 95% CI, 39 to 63). There was no overlap between these two sets of confidence intervals.

These data are broken down by drug and study number in Figure 2A. For each of the 12 drugs, the results of at least one study either were unpublished or were reported in the literature as positive despite a conflicting judgment by the FDA.

View larger version (31K): [in this window] [in a new window]   Figure 2. Publication Status and FDA Regulatory Decision by Study and by Drug. Panel A shows the publication status of individual studies. Nearly every study deemed positive by the FDA (top row) was published in a way that agreed with the FDA's judgment. By contrast, most studies deemed negative (bottom row) or questionable (middle row) by the FDA either were published in a way that conflicted with the FDA's judgment or were not published. Numbers shown in boxes indicate individual studies and correspond to the study numbers listed in Table A of the Supplementary Appendix. Panel B shows the numbers of patients participating in the individual studies indicated in Panel A. Data for patients who participated in studies deemed positive by the FDA were very likely to be published in a way that agreed with the FDA's judgment. By contrast, data for patients who participated in studies deemed negative or questionable by the FDA tended either not to be published or to be published in a way that conflicted with the FDA's judgment.

 
Number of Study Participants

As shown in Table 1, a total of 12,564 patients participated in these trials. The data from 3449 patients (27%) were not published. Data from an additional 1843 patients (15%) were reported in journal articles in which the highlighted finding conflicted with the FDA-defined primary outcome. Thus, the percentages for the patients closely mirrored those for the studies (Table 1).

Whether a patient's data were reported in a way that was in concert with the FDA review was associated with the study outcome (Figure 1B) (risk ratio, 27.1), which was consistent with the above-reported finding with the studies. Figure 2B shows these same data according to the drug being evaluated.

Qualitative Description of Selective Reporting within Trials

The methods reported in 11 journal articles appear to depart from the prespecified methods reflected in the FDA reviews (Table B of the Supplementary Appendix). Although for each of these studies the finding with respect to the protocol-specified primary outcome was nonsignificant, each publication highlighted a positive result as if it were the primary outcome. The nonsignificant results for the prespecified primary outcomes were either subordinated to nonprimary positive results (in two reports) or omitted (in nine). (Study-level methodologic differences are detailed in the footnotes to Table B of the Supplementary Appendix.)

Effect Size

The effect-size values derived from the journal reports were often greater than those derived from the FDA reviews. The difference between these two sets of values was significant whether the studies (P=0.003) or the drugs (P=0.012) were used as the units of analysis (see Table D in the Supplementary Appendix).

The effect sizes of the published and unpublished studies reviewed by the FDA are compared in Figure 3A. The overall mean weighted effect-size value was 0.37 (95% CI, 0.33 to 0.41) for published studies and 0.15 (95% CI, 0.08 to 0.22) for unpublished studies. The difference was significant whether the studies (P<0.001) or the drugs (P=0.005) were used as the units of analysis (Table D in the Supplementary Appendix).

View larger version (43K): [in this window] [in a new window]   Figure 3. Mean Weighted Effect Size According to Drug, Publication Status, and Data Source. Values for effect size are expressed as Hedges's g (the difference between two means divided by their pooled standard deviation). Effect-size values of 0.2 and 0.5 are considered to be small and medium, respectively.29 Effect-size values for unpublished studies and published studies, as extracted from data in FDA reviews, are shown in Panel A. Horizontal lines indicate 95% confidence intervals. There were no unpublished studies for controlled-release paroxetine or fluoxetine. For each of the other antidepressants, the effect size for the published subgroup of studies was greater than the effect size for the unpublished subgroup of studies. Overall effect-size values (i.e., based on data from the FDA for published and unpublished studies combined), as compared with effect-size values based on data from corresponding published reports, are shown in Panel B. For each drug, the effect-size value based on published literature was higher than the effect-size value based on FDA data, with increases ranging from 11 to 69%. For the entire drug class, effect sizes increased by 32%.

 
The mean effect-size values for all FDA studies, both published and unpublished, are compared with those for all published studies, as shown in Figure 3B. Again, the differences were significant whether the studies (P<0.001) or the drugs (P=0.002) were used as units of analysis (Table D in the Supplementary Appendix).

For each of the 12 drugs, the effect size derived from the journal articles exceeded the effect size derived from the FDA reviews (sign test, P<0.001) (Figure 3B). The magnitude of the increases in effect size between the FDA reviews and the published reports ranged from 11 to 69%, with a median increase of 32%. A 32% increase was also observed in the weighted mean effect size for all drugs combined, from 0.31 (95% CI, 0.27 to 0.35) to 0.41 (95% CI, 0.36 to 0.45).

A list of the study-level effect-size values used in the above analyses — derived from both the FDA reviews and the published reports — is provided in Table C of the Supplementary Appendix. These effect-size values are based on P values and sample sizes shown in Table A of the Supplementary Appendix, which also lists reference information for the publications consulted.

Discussion

We found a bias toward the publication of positive results. Not only were positive results more likely to be published, but studies that were not positive, in our opinion, were often published in a way that conveyed a positive outcome. We analyzed these data in terms of the proportion of positive studies and in terms of the effect size associated with drug treatment. Using both approaches, we found that the efficacy of this drug class is less than would be gleaned from an examination of the published literature alone. According to the published literature, the results of nearly all of the trials of antidepressants were positive. In contrast, FDA analysis of the trial data showed that roughly half of the trials had positive results. The statistical significance of a study's results was strongly associated with whether and how they were reported, and the association was independent of sample size. The study outcome also affected the chances that the data from a participant would be published. As a result of selective reporting, the published literature conveyed an effect size nearly one third larger than the effect size derived from the FDA data.

Previous studies have examined the risk–benefit ratio for drugs after combining data from regulatory authorities with data published in journals.^3,30,31,32 We built on this approach by comparing study-level data from the FDA with matched data from journal articles. This comparative approach allowed us to quantify the effect of selective publication on apparent drug efficacy.

Our findings have several limitations: they are restricted to antidepressants, to industry-sponsored trials registered with the FDA, and to issues of efficacy (as opposed to "real-world" effectiveness³³). This study did not account for other factors that may distort the apparent risk–benefit ratio, such as selective publication of safety issues, as has been reported with rofecoxib (Vioxx, Merck)³⁴ and with the use of selective serotonin-reuptake inhibitors for depression in children.³ Because we excluded articles covering multiple studies, we probably counted some studies as unpublished that were — technically — published. The practice of bundling negative and positive studies in a single article has been found to be associated with duplicate or multiple publication,³⁵ which may also influence the apparent risk–benefit ratio.

There can be many reasons why the results of a study are not published, and we do not know the reasons for nonpublication. Thus, we cannot determine whether the bias observed resulted from a failure to submit manuscripts on the part of authors and sponsors, decisions by journal editors and reviewers not to publish submitted manuscripts, or both.

We wish to clarify that nonsignificance in a single trial does not necessarily indicate lack of efficacy. Each drug, when subjected to meta-analysis, was shown to be superior to placebo. On the other hand, the true magnitude of each drug's superiority to placebo was less than a diligent literature review would indicate.

We do not mean to imply that the primary methods agreed on between sponsors and the FDA are necessarily preferable to alternative methods. Nevertheless, when multiple analyses are conducted, the principle of prespecification controls the rate of false positive findings (type I error), and it prevents HARKing,³⁶ or hypothesizing after the results are known.

It might be argued that some trials did not merit publication because of methodologic flaws, including problems beyond the control of the investigator. However, since the protocols were written according to international guidelines for efficacy studies³⁷ and were carried out by companies with ample financial and human resources, to be fair to the people who put themselves at risk to participate, a cogent public reason should be given for failure to publish.

Selective reporting deprives researchers of the accurate data they need to estimate effect size realistically. Inflated effect sizes lead to underestimates of the sample size required to achieve statistical significance. Underpowered studies — and selectively reported studies in general — waste resources and the contributions of investigators and study participants, and they hinder the advancement of medical knowledge. By altering the apparent risk–benefit ratio of drugs, selective publication can lead doctors to make inappropriate prescribing decisions that may not be in the best interest of their patients and, thus, the public health.

Dr. Turner reports having served as a medical reviewer for the Food and Drug Administration. No other potential conflict of interest relevant to this article was reported.

We thank Emily Kizer, Marcus Griffith, and Tammy Lewis for clerical assistance; David Wilson, Alex Sutton, Ohidul Siddiqui, and Benjamin Chan for statistical consultation; Linda Ganzini, Thomas B. Barrett, and Daniel Hilfet-Hilliker for their comments on an earlier version of this manuscript; Arifula Khan, Kelly Schwartz, and David Antonuccio for providing access to FDA reviews; Thomas B. Barrett, Norwan Moaleji and Samantha Ruimy for double data extraction and entry; and Andrew Hamilton for literature database searches.

Source Information

From the Departments of Psychiatry (E.H.T., A.M.M.) and Pharmacology (E.H.T.), Oregon Health and Science University; and the Behavioral Health and Neurosciences Division, Portland Veterans Affairs Medical Center (E.H.T., A.M.M., R.A.T.) — both in Portland, OR; the Department of Psychology, Kent State University, Kent, OH (E.L.); the Department of Psychology, University of California–Riverside, Riverside (R.R.); and Harvard University, Cambridge, MA (R.R.).

Address reprint requests to Dr. Turner at Portland VA Medical Center, P3MHDC, 3710 SW US Veterans Hospital Rd., Portland, OR 97239, or at turnere{at}ohsu.edu.

References

1. Hagdrup N, Falshaw M, Gray RW, Carter Y. All members of primary care team are aware of importance of evidence based medicine. BMJ 1998;317:282-282. [Free Full Text]
2. Craig JC, Irwig LM, Stockler MR. Evidence-based medicine: useful tools for decision making. Med J Aust 2001;174:248-253. [Web of Science][Medline]
3. Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet 2004;363:1341-1345. [CrossRef][Web of Science][Medline]
4. Kyzas PA, Loizou KT, Ioannidis JP. Selective reporting biases in cancer prognostic factor studies. J Natl Cancer Inst 2005;97:1043-1055. [Free Full Text]
5. Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr. Publication bias and clinical trials. Control Clin Trials 1987;8:343-353. [CrossRef][Web of Science][Medline]
6. Simes RJ. Confronting publication bias: a cohort design for meta-analysis. Stat Med 1987;6:11-29. [Web of Science][Medline]
7. Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315:640-645. [Free Full Text]
8. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291:2457-2465. [Free Full Text]
9. Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279:281-286. [Free Full Text]
10. Chan AW, Krleza-Jeri K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 2004;171:735-740. [Free Full Text]
11. Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ 2005;330:753-753. [Free Full Text]
12. Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ 2006;333:597-600. [Free Full Text]
13. Hayashino Y, Noguchi Y, Fukui T. Systematic evaluation and comparison of statistical tests for publication bias. J Epidemiol 2005;15:235-243. [CrossRef][Web of Science][Medline]
14. Pham B, Platt R, McAuley L, Klassen TP, Moher D. Is there a "best" way to detect and minimize publication bias? An empirical evaluation. Eval Health Prof 2001;24:109-125. [Free Full Text]
15. Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol 2000;53:1119-1129. [CrossRef][Web of Science][Medline]
16. Ioannidis JP, Trikalinos TA. The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ 2007;176:1091-1096. [Free Full Text]
17. Turner EHA. A taxpayer-funded clinical trials registry and results database. PLoS Med 2004;1:e60-e60. [CrossRef][Medline]
18. Center for Drug Evaluation and Research. Manual of policies and procedures: clinical review template. Rockville, MD: Food and Drug Administration, 2004. (Accessed December 20, 2007, at http://www.fda.gov/cder/mapp/6010.3.pdf.)
19. Committee on Government Reform, U.S. House of Representatives, 109th Congress, 1st Session. A citizen's guide on using the Freedom of Information Act and the Privacy Act of 1974 to request government records. Report no. 109-226. Washington, DC: Government Printing Office, 2005. (Also available at: http://www.fas.org/sgp/foia/citizen.pdf.)
20. Center for Drug Evaluation and Research. Drugs@FDA: FDA approved drug products. Rockville, MD: Food and Drug Administration. (Accessed December 20, 2007, at http://www.accessdata.fda.gov/scripts/cder/drugsatfda.)
21. International Conference on Harmonisation (ICH), European Medicines Agency (EMEA). Topic E9: statistical principles for clinical trials. Rockville, MD: Food and Drug Administration. (Accessed December 20, 2007, at http://www.fda.gov/cder/guidance/iche3.pdf.)
22. Temple R, Ellenberg SS. Placebo-controlled trials and active-control trials in the evaluation of new treatments. Part 1: ethical and scientific issues. Ann Intern Med 2000;133:455-463. [Free Full Text]
23. Hedges LV. Estimation of effect size from a series of independent experiments. Psychol Bull 1982;92:490-499. [CrossRef][Web of Science]
24. Rosenthal R. Meta-analytic procedures for social research. Newbury Park, CA: Sage, 1991.
25. Whitley E, Ball J. Statistics review 5: comparison of means. Crit Care 2002;6:424-428. [CrossRef][Web of Science][Medline]
26. Hedges LV, Olkin I. Statistical methods for meta-analysis. New York: Academic Press, 1985.
27. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986;7:177-188. [CrossRef][Web of Science][Medline]
28. Stata statistical software, release 9. College Station, TX: StataCorp, 2005.
29. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. New York: Lawrence Erlbaum Associates, 1988.
30. Nissen SE, Wolski K. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. N Engl J Med 2007;356:2457-2471. [Erratum, N Engl J Med 2007;357:100.] [Free Full Text]
31. Nissen SE, Wolski K, Topol EJ. Effect of muraglitazar on death and major adverse cardiovascular events in patients with type 2 diabetes mellitus. JAMA 2005;294:2581-2586. [Free Full Text]
32. Sackner-Bernstein JD, Kowalski M, Fox M, Aaronson K. Short-term risk of death after treatment with nesiritide for decompensated heart failure: a pooled analysis of randomized controlled trials. JAMA 2005;293:1900-1905. [Free Full Text]
33. Revicki DA, Frank L. Pharmacoeconomic evaluation in the real world: effectiveness versus efficacy studies. Pharmacoeconomics 1999;15:423-434. [CrossRef][Web of Science][Medline]
34. Topol EJ. Failing the public health -- rofecoxib, Merck, and the FDA. N Engl J Med 2004;351:1707-1709. [Free Full Text]
35. Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine -- selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ 2003;326:1171-1173. [Free Full Text]
36. Kerr NL. HARKing: hypothesizing after the results are known. Pers Soc Psychol Rev 1998;2:196-217. [Free Full Text]
37. International Conference on Harmonisation — Efficacy. Rockville, MD: Food and Drug Administration. (Accessed December 20, 2007, at http://www.fda.gov/cder/guidance/#ICH_efficacy.)

Abstract PDF PDA Full Text PowerPoint Slide Set Supplementary Material Letters Add to Personal Archive Add to Citation Manager Notify a Friend E-mail When Cited E-mail When Letters Appear PubMed Citation

Related Letters:

Selective Publication of Antidepressant Trials
de Jonge P., Bockting C. L., Schoones J. W., Ninan P. T., Poole R. M., Stiles G. L., Turner E. H., Tell R. A.
Extract | Full Text | PDF  
N Engl J Med 2008; 358:2180-2182, May 15, 2008. Correspondence

This article has been cited by other articles:

• Vedula, S. S., Bero, L., Scherer, R. W., Dickersin, K. (2009). Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use. NEJM 361: 1963-1971 [Abstract] [Full Text]  
• Will, E. J. (2009). Caveats for Scientific Publication in the Modern Marketplace. CJASN 4: 1693-1695 [Full Text]  
• Kelley, J. M., Lembo, A. J., Ablon, J. S., Villanueva, J. J., Conboy, L. A., Levy, R., Marci, C. D., Kerr, C. E., Kirsch, I., Jacobson, E. E., Riess, H., Kaptchuk, T. J. (2009). Patient and Practitioner Influences on the Placebo Effect in Irritable Bowel Syndrome. Psychosom. Med. 71: 789-797 [Abstract] [Full Text]  
• Pandolfini, C., Bonati, M., Sammons, H. M (2009). Registration of trials in children: update of current international initiatives. Arch. Dis. Child. 94: 717-719 [Full Text]  
• Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P.A., Clarke, M., Devereaux, P. J., Kleijnen, J., Moher, D. (2009). The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. ANN INTERN MED 151: W-65-W-94 [Abstract] [Full Text]  
• Moreno, S. G, Sutton, A. J, Turner, E. H, Abrams, K. R, Cooper, N. J, Palmer, T. M, Ades, A E (2009). Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. BMJ 339: b2981-b2981 [Abstract] [Full Text]  
• Steel, Z., Chey, T., Silove, D., Marnane, C., Bryant, R. A., van Ommeren, M. (2009). Association of Torture and Other Potentially Traumatic Events With Mental Health Outcomes Among Populations Exposed to Mass Conflict and Displacement: A Systematic Review and Meta-analysis. JAMA 302: 537-549 [Abstract] [Full Text]  
• Abraham, J., Davis, C. (2009). Drug evaluation and the permissive principle: continuities and contradictions between standards and practices in antidepressant regulation.. Social Studies of Science 39: 569-598 [Abstract]  
• Ghaemi, S N (2009). The failure to know what isn't known: negative publication bias with lamotrigine and a glimpse inside peer review. Evid. Based Ment. Health 12: 65-68 [Full Text]  
• Olfson, M., Marcus, S. C. (2009). National Patterns in Antidepressant Medication Treatment. Arch Gen Psychiatry 66: 848-856 [Abstract] [Full Text]  
• Liberati, A., Altman, D. G, Tetzlaff, J., Mulrow, C., Gotzsche, P. C, Ioannidis, J. P A, Clarke, M., Devereaux, P J, Kleijnen, J., Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 339: b2700-b2700 [Abstract] [Full Text]  
• O'Connor, A. B. (2009). The Need for Improved Access to FDA Reviews. JAMA 302: 191-193 [Full Text]  
• Strom, M., Mortensen, E. L, Halldorsson, T. I, Thorsdottir, I., Olsen, S. F (2009). Fish and long-chain n-3 polyunsaturated fatty acid intakes during pregnancy and risk of postpartum depression: a prospective study based on a large national birth cohort. Am. J. Clin. Nutr. 90: 149-155 [Abstract] [Full Text]  
• Louhiala, P (2009). Bone marrow transplantation in the prevention of intellectual disability due to inherited metabolic disease: ethical issues. J. Med. Ethics 35: 415-418 [Abstract] [Full Text]  
• Claes, S. (2009). Targeting the HPA axis in major depression: does it work?: INVITED COMMENTARY ON... ANTIGLUCOCORTICOIDS IN PSYCHIATRY. Adv. Psychiatr. Treat. 15: 250-252 [Abstract] [Full Text]  
• Guo, S.-W., Hummelshoj, L., Olive, D. L., Bulun, S. E., D'Hooghe, T. M., Evers, J. L.H. (2009). A call for more transparency of registered clinical trials on endometriosis. Hum Reprod 24: 1247-1254 [Abstract] [Full Text]  
• Antonuccio, D., Healy, D. (2009). Stealth advertising and academic stalking. BMJ 338: b1612-b1612 [Full Text]  
• Psaty, B. M. (2009). Conflict of Interest, Disclosure, and Trial Reports. JAMA 301: 1477-1479 [Full Text]  
• Sismondo, S. (2009). Ghosts in the machine: publication planning in the medical sciences.. Social Studies of Science 39: 171-198 [Abstract]  
• Carney, R. M., Freedland, K. E. (2009). Treatment-Resistant Depression and Mortality After Acute Coronary Syndrome. Am. J. Psychiatry 166: 410-417 [Abstract] [Full Text]  
• Whang, W., Kubzansky, L. D., Kawachi, I., Rexrode, K. M., Kroenke, C. H., Glynn, R. J., Garan, H., Albert, C. M. (2009). Depression and risk of sudden cardiac death and coronary heart disease in women: results from the Nurses' Health Study.. J Am Coll Cardiol 53: 950-958 [Abstract] [Full Text]  
• van't Veer-Tazelaar, P. J., van Marwijk, H. W. J., van Oppen, P., van Hout, H. P. J., van der Horst, H. E., Cuijpers, P., Smit, F., Beekman, A. T. F. (2009). Stepped-Care Prevention of Anxiety and Depression in Late Life: A Randomized Controlled Trial. Arch Gen Psychiatry 66: 297-304 [Abstract] [Full Text]  
• Wood, A. J.J. (2009). Progress and Deficiencies in the Registration of Clinical Trials. NEJM 360: 824-830 [Full Text]  
• Angell, M. (2009). Relationships with the drug industry: Keep at arm's length. BMJ 338: b222-b222 [Full Text]  
• Mathew, S. J., Charney, D. S. (2009). Publication Bias and the Efficacy of Antidepressants. Am. J. Psychiatry 166: 140-145 [Full Text]  
• Rousseau, P. (2009). Evidence-based Medicine: Show Me the Evidence!. AM J HOSP PALLIAT CARE 26: 5-7  
• Bridge, J. A., Birmaher, B., Iyengar, S., Barbe, R. P., Brent, D. A. (2009). Placebo Response in Randomized Controlled Trials of Antidepressants for Pediatric Major Depressive Disorder. Am. J. Psychiatry 166: 42-49 [Abstract] [Full Text]  
• Chan, A.-W., Hrobjartsson, A., Jorgensen, K. J, Gotzsche, P. C, Altman, D. G (2008). Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols. BMJ 337: a2299-a2299 [Abstract] [Full Text]  
• Wittkampf, K A, van Zwieten, M, Smits, F T., Schene, A H, Huyser, J, van Weert, H C (2008). Patients' view on screening for depression in general practice. Fam Pract 25: 438-444 [Abstract] [Full Text]  
• Thombs, B. D., de Jonge, P., Coyne, J. C., Whooley, M. A., Frasure-Smith, N., Mitchell, A. J., Zuidersma, M., Eze-Nliam, C., Lima, B. B., Smith, C. G., Soderlund, K., Ziegelstein, R. C. (2008). Depression Screening and Patient Outcomes in Cardiovascular Care: A Systematic Review. JAMA 300: 2161-2171 [Abstract] [Full Text]  
• Hansen, R., Gaynes, B., Thieda, P., Gartlehner, G., Deveaugh-Geiss, A., Krebs, E., Lohr, K. (2008). Meta-analysis of Major Depressive Disorder Relapse and Recurrence With Second-Generation Antidepressants. Psychiatr. Serv. 59: 1121-1130 [Abstract] [Full Text]  
• Angell, M. (2008). Industry-Sponsored Clinical Research: A Broken System. JAMA 300: 1069-1071 [Full Text]  
• Ramsey, S., Scoggins, J. (2008). Commentary: Practicing on the Tip of an Information Iceberg? Evidence of Underpublication of Registered Clinical Trials in Oncology. The Oncologist 13: 925-929 [Abstract] [Full Text]  
• Brown, G.S., Cameron, J., Brown, L. (2008). In Search of the Active Ingredient: What Really Works in Mental Health Care?. Fluency and Fluency Disorders 18: 53-59 [Abstract] [Full Text]  
• Abel, K. M (2008). Review: Psychosocial and psychological interventions reduce postpartum depressive symptoms. Evid. Based Ment. Health 11: 79-79 [Full Text]  
• Troy, D. E., Gottlieb, S., Kesselheim, A. S., Avorn, J. (2008). Pharmaceutical Promotion and First Amendment Rights. NEJM 359: 536-537 [Full Text]  
• Furman, L. M. (2008). Attention-Deficit Hyperactivity Disorder (ADHD): Does New Research Support Old Concepts?. J Child Neurol 23: 775-784 [Abstract]  
• Frasure-Smith, N., Lesperance, F. (2008). Heterogeneity of Patients With Coronary Artery Disease and Distress and the Need to Identify Relevant Subtypes--Reply. Arch Gen Psychiatry 65: 852-853 [Full Text]  
• Rifai, N., Altman, D. G., Bossuyt, P. M. (2008). Reporting Bias in Diagnostic and Prognostic Studies: Time for Action. Clin. Chem. 54: 1101-1103 [Full Text]  
• Fava, G. A (2008). Should the drug industry work with key opinion leaders? No. BMJ 336: 1405-1405 [Full Text]  
• WALKER, E., HERNANDEZ, A. V., KATTAN, M. W. (2008). Meta-analysis: Its strengths and limitations. Cleveland Clinic Journal of Medicine 75: 431-439 [Abstract] [Full Text]  
• de Jonge, P., Bockting, C. L., Schoones, J. W., Ninan, P. T., Poole, R. M., Stiles, G. L., Turner, E. H., Tell, R. A. (2008). Selective Publication of Antidepressant Trials. NEJM 358: 2180-2182 [Full Text]  
• Deshauer, D. MD MSc, Moher, D. PhD, Fergusson, D. PhD, Moher, E. BA, Sampson, M. MLIS, Grimshaw, J. MD PhD (2008). Selective serotonin reuptake inhibitors for unipolar depression: a systematic review of classic long-term randomized controlled trials. CMAJ 178: 1293-1301 [Abstract] [Full Text]  
• Hackbarth, D. (2008). Research Reporting and Evidence of Effectiveness: Why "No Difference" Matters. Am J Crit Care 17: 218-220 [Full Text]  
• Psaty, B. M., Ray, W. (2008). FDA Guidance on Off-Label Promotion and the State of the Literature From Sponsors. JAMA 299: 1949-1951 [Full Text]  
• Psaty, B. M., Kronmal, R. A. (2008). Reporting Mortality Findings in Trials of Rofecoxib for Alzheimer Disease or Cognitive Impairment: A Case Study Based on Documents From Rofecoxib Litigation. JAMA 299: 1813-1817 [Abstract] [Full Text]  
• Mamdani, M. M. PharmD MPH (2008). Health advisories: when good intentions go bad. CMAJ 178: 1025-1026 [Full Text]  
• Antonuccio, D. O, Healy, D. (2008). The researcher's credo. BMJ 336: 629-629 [Full Text]  
• Lenzer, J., Brownlee, S. (2008). An untold story?. BMJ 336: 532-534 [Full Text]  
• Turner, E. H, Rosenthal, R. (2008). Efficacy of antidepressants. BMJ 336: 516-517 [Full Text]  
• Greenland, P., Lloyd-Jones, D. (2008). Critical Lessons From the ENHANCE Trial. JAMA 299: 953-955 [Full Text]  
• (2008). Selective Publication of Positive Drug Trials. JWatch General 2008: 9-9 [Full Text]  
• Groves, T. (2008). Mandatory disclosure of trial results for drugs and devices. BMJ 336: 170-170 [Full Text]  
• (2008). All you need to read in the other general journals. BMJ 336: 182-183 [Full Text]