Skip to content

Advertisement

Open Access

Reporting quality of randomized controlled trials in otolaryngology: review of adherence to the CONSORT statement

Journal of Otolaryngology - Head & Neck Surgery201847:34

https://doi.org/10.1186/s40463-018-0277-8

Received: 10 November 2017

Accepted: 29 April 2018

Published: 15 May 2018

Abstract

Background

Randomized controlled trials are the gold standard in medical and surgical research to assess the efficacy of therapeutic interventions. The reporting of these trials should be of high quality to allow readers’ appropriate interpretation and application.

Methods

The objectives of our study were to assess the extent to which the recent Otolaryngology – Head and Neck Surgery (ORL-HNS) randomized control trials in the top nine journals and in the top Canadian journal comply with the Consolidated Standards of Reporting Trials (CONSORT) statement, and to identify the CONSORT items most in need of improvement. Based on the impact factor and circulation number of 2014, the top nine Otolaryngology journals and the top Canadian Otolaryngology journal were selected and were searched to identify RCTs published in English and between 2010 and 2014. Two authors independently reviewed and extracted data using a standardized data extraction form constructed with the help of a medical librarian. Our outcome was to assess the adherence of articles reporting to the CONSORT items. Descriptive statistics were used.

Results

One hundred and eighty-two Otolaryngologic RCTs were identified in the top nine international journals and in the top Canadian journal. The inter-rater reliability between two raters was 0.32. The extent of adherence to CONSORT Statement ranged from 25 to 93.5% with a mean of 59.0% and a median of 59.4%. Only 6.5% of RCTs described the individual responsible for enrolling and assigning subjects and method of randomization; 32.4% reported the estimated effect size and precision; 40.6% reported a sample size calculation and 32.4% mentioned external validity or implications of the findings.

Conclusion

Findings revealed that the reporting of RCTs in the top nine ORL-HNS journals and in the top Canadian ORL-HNS journal is suboptimal. The quality of reporting can be improved by addressing the three CONSORT items found most deficient in this study namely, sample size calculations, estimated effect size and precision, and external validity.

Keywords

CONSORT adherenceRCTs in otolaryngologyReporting quality

Background

Randomized Controlled Trials (RCTs) are the preferred study design for comparing therapeutic interventions in medicine; they are considered the cornerstone of evidence-based medicine. However, poor reporting of RCTs impedes adequate understanding of the clinical indications. Readers require clear, transparent and complete information to assess the quality and results of a trial. Because biases can occur in all aspects of studies, poor reporting limits the reader’s appreciation of the result’s validity [1]. Flawed reporting that omits important methodological details further prevents their incorporation in systematic reviews and meta-analyses [2]. To improve clarity and transparency of reporting of RCTs, the Consolidated Standards of Reporting Trials (CONSORT) statement was released in 1996 and revised in 2001 and in 2010. The CONSORT Statement and the corresponding checklist summarize the essential items that should be reported.

The quality of reporting of RCTs in surgery is inferior to that in medicine [3]. Important differences in RCTs implementation exist in surgical disciplines from the medical disciplines. Challenges in fulfilling the criteria for RCTs including blinding and the creation of placebo patients have led to suboptimal quality of reporting, as demonstrated in a study across six surgical specialties [4]. In ORL-HNS, this shortcoming has been especially noted by Ah-See et al. in 1998, where they analyzed RCTs published over 30-year period (1966-1995) and concluded that the quality of reporting in the domain was unsatisfactory [5]. The CONSORT Non Pharmacological Treatments (CONSORT-NPT) was released with the goal to remediate to the poor adherence to the CONSORT checklist of RCTs in surgical specialties. Nevertheless, recent similar studies assessing the quality of reporting of RCTs in General Surgery and Plastic Surgery have revealed even poorer adherence to CONSORT-NPT compared to the standard CONSORT checklist [3, 4]. With the CONSORT checklist updated in 2010, there are great hopes that publications in surgical specialties will have improved RCTs reporting. Most recently, Peters et al. scored 18 articles published in ORL-HNS journals reporting a mean score of 71.8% with a significant lower grade to general medical journals [6]. To date, there have been no studies of a large number of RCTs investigating the compliance of ORL-HNS to the 2010 version of CONSORT checklist. Our primary outcomes were to evaluate the adherence to the CONSORT checklist during the period 2011-2014 and identify the items most in need of amelioration. Specifically, we recorded the adherence of RCTs to items of the CONSORT 2010 checklist to determine the progression in quality of reporting in ORL-HNS compared to the assessment of Ah-See et al. conducted 16 years ago [5].

Methods

This study is exempted from institutional board review as all articles were publicly available.

Selection of ORL-HNS journals

On a review of the highest-ranking impact factor international ORL-HNS journals of 2014 with the highest number of circulation, top nine journals were chosen, and one top Canadian otolaryngology journal was included in our study to add on the national perspective.

Search method

With the assistance of a medical librarian at a tertiary centre, we performed a structured search of the MEDLINE database to identify all RCTs published in the top nine journals and in the top Canadian journal between January 1, 2011, and June 4, 2014, corresponding to the CONSORT 2010 update. Each title and abstract from the search resulted articles were screened for inclusion and exclusion criteria. Studies describing interventions performed on human subjects and written in English language were included. Exclusion criteria were animal studies, reviews, and non-RCTs. All references satisfying the inclusion criteria were further screened to ensure that the study fulfilled our search requirements.

Rater training

All included articles were read in-depth either by the first or the second author. To assure inter-observer concordance, the reviewers (YQH and KT) were trained to first score separately the same five RCTs. Both their results were compared and verified by another senior reviewer (BI). Then, ten randomly selected studies were evaluated separately by all three reviewers and, compared with the results of an epidemiologist (MJS). Meanings and interpretations of all CONSORT criteria were discussed, and the consensus was reached among reviewers where discrepancies existed. Finally, the two reviewers (YQH and KT) each read exhaustively and scored each independently half of the remaining RCTs sorted by alphabetical order of the first author’s name.

Statistical analyses

Descriptive statistics consisting of frequencies and percentages calculations were used to portray characteristics of our series.

Results

The inter-rater reliability between the two raters (YQH and KT) was 0.32 using Cohen’s Kappa with observed agreement of 0.87. The total number of RCTs identified by title or abstract was 467 (Fig. 1). Twenty of the articles were duplicates, and 265 did not meet inclusion criteria. The remaining 182 RCTs which came from eight different journals (Table 1) were read in full by either YQH or KT. Most articles were found in the Otolaryngology - Head and Neck Surgery (25.8% of all trials) and in the Laryngoscope (32.3% of all trials).
Figure 1
Fig. 1

Flow Diagram of Article Selections

Table 1

Selected Journals in ORL-HNS

Journal Name

(Country)

Laryngoscopea(American)

Archives of Otolaryngology-Head and Neck Surgery

=JAMA Otolaryngology - Head And Neck Surgerya(American)

Annals of Otology, Rhinology and Laryngology(American)

Otolaryngology - Head and Neck Surgerya(American)

Current Opinion In Otolaryngology And Head And Neck Surgery (British)

Journal of Otolaryngology - Head and Neck Surgery

(Canadian)

American Journal of Otolaryngology

(American)

Otolaryngologic Clinics of North America

(American)

Journal of Laryngology and Otology

(British)

Clinical Otolaryngology

(British)a

Extent of adherence to the CONSORT statement

The extent of adherence to the CONSORT Statement is the percentage of CONSORT items reported in the article. The mean extent of adherence found for 182 RCTs included in our study was 59.0%, with a range 25 to 93.5% and a median of 59.4%.

CONSORT checklist items

To assess the quality of reporting, the most recent version of the CONSORT Statement was used. All CONSORT items and the frequency of adherence to the individual criterion are shown in Table 2. The items that were described in more than 90% of articles are the report of eligibility criteria of participants (92.3%), of intervention (99.5%), of used statistical methods (96.7%), of number of participants throughout different steps of the study (94.0%), of included patients number (90.1%), and of interpretation (96.2%). The items that were described in less than 50% of trials are the reporting of primary and secondary outcomes (42.3%), sample size calculation (40.6%), person in charge of randomization, allocation and assignment of participants (6.6%), effect size (32.4%), and generalizability (32.4%).
Table 2

Frequency and Percentage of Adherence to Individual Criterion of the CONSORT 2010 Checklist

Criterion

CONSORT item

Frequency

%

1a

Identification as a randomized trial in the title

80

44.0%

1b

Structured summary of trial design, methods, results, and conclusions

5

2.7%

2a

Scientific background and explanation of rationale

182

100%

2b

Specific objectives or hypotheses

182

100%

3a

Description of trial design (such as parallel, factorial) including allocation ratio

182

100%

3b

Important changes to methods after trial commencement (such as eligibility criteria), with reasons

3

100%

4a

Eligibility criteria for participants

168

92.3%

4b

Settings and locations where the data were collected

140

76.9

5

The interventions for each group with sufficient details to allow replication, including how and when they were actually administered

181

99.5%

6a

Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed

77

42.3%

6b

Any changes to trial outcomes after the trial commenced, with reasons

1

100%

7a

How sample size was determined

74

40.6%

7b

When applicable, explanation of any interim analyses and stopping guidelines

2

100%

8a

Method used to generate the random allocation sequence

106

58.6%

8b

Type of randomization; details of any restriction (such as blocking and block size)

45

24.7%

9

Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned

113

62.1%

10

Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions

12

6.6%

11a

If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how

108

59.7%

11b

If relevant, description of the similarity of interventions

54

32.3%

12a

Statistical methods used to compare groups for primary and secondary outcomes

176

96.7%

12b

Methods for additional analyses, such as subgroup analyses and adjusted analyses

23

88.5%

13a

For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analyzed for the primary outcome

171

94.0%

13b

For each group, losses and exclusions after randomization, together with reasons

146

80.2%

14a

Dates defining the periods of recruitment and follow-up

96

52.7%

14b

Why the trial ended or was stopped

1

100%

15

A table showing baseline demographic and clinical characteristics for each group

132

72.5%

16

For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups

164

90.1%

17a

For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval)

59

32.4%

17b

For binary outcomes, presentation of both absolute and relative effect sizes is recommended

13

41.9%

18

Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre-specified from exploratory

7

6.5%

19

All important harms or unintended effects in each group (for specific guidance see CONSORT for harms)

138

78.0%

20

Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses

109

59.9%

21

Generalizability (external validity, applicability) of the trial findings

59

32.4%

22

Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence

175

96.2%

23

Registration number and name of trial registry

19

10.4%

24

Where the full trial protocol can be accessed, if available

7

3.8%

25

Sources of funding and other support (such as supply of drugs), role of funders

156

85.7%

Discussion

Our study revealed an overall mean adherence to the CONSORT 2010 checklist of 59.0% in a total of 182 surgical RCTs published in the top nine ORL-HNS journals and in the top Canadian ORL-HNS. No article satisfied all criteria evaluated in the study. To our knowledge, this is the first study to review a very large number of RCTs to assess the compliance of ORL-HNS literature in all its subspecialities to the 2010 CONSORT checklist.

Our results are comparable to those published by Ah-See et al. in 1998. They showed a mean score of 7.3/12 criteria (60.8%) in 295 articles analyzed from a scoring system derived from the CONSORT Statement published in 1996. The adherence rate in the present study (59.0%), reflects the dearth of improvement in RCT reporting in ORL-HNS literature, despite multiple important updates on CONSORT guidelines since its first publication. Ifeacho et al. found that RCTs relating to adenotonsillectomy in ORL-HNS had substandard quality of reporting with 51-60% adherence to CONSORT using the 2010 checklist [7]. Carlton et al., by using the same checklist, evaluated 38 RCTs on surgical procedures in head and neck oncology surgery and found a mean checklist score of 45.4% [8]. Our results support and widen the applicability of Ifeacho et al. and of Carlton et al. findings by expanding them to the entire scope of ORL-HNS literature by including all its subspecialities and thus having a larger search return.

By looking at the scores of individual items in the CONSORT 2010 checklist, we observe that the criteria pertaining to the background section are reported. A hundred percent of trials contained scientific background, specific objectives and description. Furthermore, items that were best reported in the present study were elements from abstracts. Authors are more careful and unlikely to miss key elements of abstracts as they represent the most condensed form of the message conveyed by a study. Contrastingly, the full paper provides the opportunity to address more in depth essential details that clinicians may not feel comfortable addressing (e.g. sample size calculation) and hence do not report them precisely. Interestingly, however, Knobloch et al. have found that abstract reporting of surgical RCTs has also been plagued by suboptimal adherence to CONSORT guidelines [9]. This raises a question regarding the role that peer-reviewed journals can have in further promoting and enforcing CONSORT guidelines.

The lowest-ranking items were from the methodology, results and discussion sections. With only 40.6% of trials reporting on sample size calculations, we believe that the readership could have legitimate concerns regarding the statistical significance and robustness of the results put forward by a vast majority of RCTs examined in the present study. Indeed, sample size calculation allows the reader to independently assess and validate the power of the study. As a larger sample size provides the best guarantee to decrease both type I and type II errors it is critical for a reader to understand the process by which a certain sample size has been determined in order to adequately detect significant variations in different treatment groups. The omission may trigger questions of statistical integrity amongst readers [10, 11]. In order to improve the design and ultimately the reporting of such intricate, yet essential, parts of an RCT we recommend that a statistician and/or epidemiologist be included in the research team supervising RCTs. Diaz-Ordaz et al., as well as another study performed by our group (unpublished data) support the concept of the multidisciplinary approach to reporting RCTs. These studies have shown that the inclusion of an epidemiologist/statistician in the author list correlated with a higher propensity of reporting sample size calculations [12].

The present study demonstrates that only 32.4% of trials report the estimated effect size and precision (e.g., 95% CI). While p-values are often looked at for statistical significance, a full appreciation of the magnitude of the phenomenon observed is only possible by providing the effect size [13].

Lastly, our study shows that 32.4% of included studies reported on external validity. This information is key for readers to evaluate the applicability of the trial’s results in the reader’s respective context. Evidence suggests that quality of reporting correlates with better quality in the conduct of the trial [14]. Nevertheless, achieving full marks on the CONSORT statement does not guarantee high quality or clinical relevance.

Our study has several limitations. First, the included RCTs were divided in two and were assessed independently by each reviewer. However, we included a training period where inter-rater variability was proven to be quasi-null and a third reviewer in equivocal cases. Generalizability of the findings may be limited as articles were written in the English language and found in the MEDLINE database. We acknowledge, more importantly, the large breadth of otolaryngology publications outside of otolaryngology journals, rendering our study ineligible to reflect the reporting quality of the entire publishing otolaryngology community. Indeed, we are aware that our findings only present the reporting quality of RCTs selected based on restricting criteria (10 Otolaryngologic journals, specific period of time).

Conclusion

In conclusion, this study shows that the quality of reporting of RCTs in the top nine ORL-HNS journals and in the top Canadian journal can be improved, specifically in the areas of methodology, results and discussions of papers. The CONSORT statement promotes standardization of reporting of trials in the literature and should be adopted by authors for writing reports of trials. Its use should be encouraged by journal editors and peer reviewers. More transparent trial reports lead to improved critical appraisal and high-quality evidence-based medicine in ORL-HNS.

Notes

Abbreviations

CONSORT: 

Consolidated Standards of Reporting Trials

CONSORT-NPT: 

Consolidated Standards of Reporting Trials - Non Pharmacological Treatments

ORL-HNS: 

Otolaryngology

Head and Neck Surgery

RCTs: 

Randomized Controlled Trials

Declarations

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Authors’ contributions

YQH, KT, BD, MS and LN all participated in the study design. YQH and KT acquired the initial database with the help of a medical librarian. YQH and KT were trained by the epidemiological expert MS. YQH and KT each read exhaustively 96 final articles extracted from the top nine journals and from the top Canadian journals and rated them diligently following the CONSORT checklist. YQH and KT analysed and interpreted the data with the epidemiological support of MS. The data analysis and interpretation were also reviewed by BD. YQH drafted the manuscript and revised it following the thorough review of KT, BD, MS and LN. All authors read and approved the final manuscript, accountable for all aspects of the work in ensuring that questions related to the accuracy and integrity are appropriately invested.

Ethics approval and consent to participate

Not applicable

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Faculty of Medicine, McGill University, Montreal, Canada
(2)
Department of Otolaryngology – Head and Neck Surgery, University of Montreal, Montreal, Canada
(3)
Department of Otolaryngology – Head and Neck Surgery, McGill University, Montreal, Canada
(4)
Division of Clinical Epidemiology, Research Institute of the McGill University Health Centre, Montreal, Canada
(5)
Center for Medical Education, McGill University, Montreal, Canada
(6)
Department of ORL-HNS, Montreal Children’s Hospital, Montréal, Canada

References

  1. Stanley K. Design of Randomized Control Trials. Circulation. 2007;115:1164–9.View ArticlePubMedGoogle Scholar
  2. Juni P, et al. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ. 2001;323(7303):42–6.View ArticlePubMedPubMed CentralGoogle Scholar
  3. Nagendran M, Harding D, Teo W, et al. Poor adherence of randomised trials in surgery to CONSORT guidelines for non-pharmacological treatments (NPT): a cross-sectional study. BMJ. 2013;3(12):e003898.Google Scholar
  4. Agha RA, et al. Randomised controlled trials in plastic surgery: a systematic review of reporting quality. Eur J Plast Surg. 2014;37:55–62.View ArticlePubMedGoogle Scholar
  5. Ah-See KW, Molony NC. A qualitative assessment of randomized controlled trials in otolaryngology. J Laryngol Otol. 1998;112(5):460–3.View ArticlePubMedGoogle Scholar
  6. Peters JP, et al. Assessment of the quality of reporting of randomised controlled trials in otorhinolaryngologic literature - adherence to the CONSORT statement. PLoS One. 2015;10(3):e0122328.View ArticlePubMedPubMed CentralGoogle Scholar
  7. Ifeacho S, et al. Are randomised controlled trials involving adenotonsillectomy well reported? Int J Pediatr Otorhinolaryngol. 2011;75(7):939–42.View ArticlePubMedGoogle Scholar
  8. Carlton DA, Kocherginsky M, Langerman AJ. A systematic review of the quality of randomized controlled trials in head and neck oncology surgery. Laryngoscope. 2015 Jan;125(1):146–52.View ArticlePubMedGoogle Scholar
  9. Knobloch K, et al. Quality of reporting according to the CONSORT, STROBE and Timmer instrument at the American burn association (ABA) annual meetings 2000 and 2008. BMC Med Res Methodol. 2011;11:161.View ArticlePubMedPubMed CentralGoogle Scholar
  10. Greene T. Randomized controlled trials 5: determining the sample size and power for clinical trials and cohort studies. Methods Mol Biol. 2015;1281:225–47.View ArticlePubMedGoogle Scholar
  11. Abdulatif M, et al. Pitfalls in reporting sample size calculation in randomized controlled trials published in leading anaesthesia journals: a systematic review. Br J Anaesth. 2015;115(5):699–707.View ArticlePubMedGoogle Scholar
  12. Diaz-Ordaz K, et al. A systematic review of cluster randomised trials in residential facilities for older people suggests how to improve quality. BMC Med Res Methodol. 2013;13:127.View ArticlePubMedPubMed CentralGoogle Scholar
  13. Parfrey PS. Randomized controlled trials 6: on contamination and estimating the actual treatment effect. Methods Mol Biol. 2015;1281:249–59.View ArticlePubMedGoogle Scholar
  14. Bryant TN. The presentation of statistics. Pediatr Allergy Immunol. 1998;9(3):108–15.View ArticlePubMedGoogle Scholar

Copyright

© The Author(s). 2018

Advertisement