The Quality of Research is Important

Type of Manuscript: Review

Title: The Quality of Research is Important

Running Title: Research Quality

Authors:

1. Rijo Mathew Choorakuttil, Research Action Group of AMMA Education Research Foundation, AMMA Center for Preventive and Diagnostic Medicine, Kochi, India. E-mail: erfamma@gmail.com
2. Sharon Baisil, MD, Research Action Group of AMMA Education Research Foundation, AMMA Center for Preventive and Diagnostic Medicine, Kochi, India. E-mail: erfamma@gmail.com
3. Praveen K Nirmalan, MPH, Research Action Group of AMMA Education Research Foundation, AMMA Center for Preventive and Diagnostic Medicine, Kochi, India. E-mail: erfamma@gmail.com

Disclosures: There are no conflicts of interest or financial interests to declare pertaining to this manuscript

Word Count: Abstract-133

Word Count: Manuscript (excluding references and abstract):1,779

Abstract

Research output and journals publishing research is increasing and has led to a lot of information made available to clinicians and the general public. While the availability of information is laudable, information of questionable quality can lead to unintended disastrous consequences. Filtering out information for quality is a herculean task and it is possible that poor quality information may dominate good quality information just because of a misbalance in the number of research studies. Researchers do not set out to do research of poor quality and it is important to be aware of several factors that may affect the quality of research. An understanding of some, if not all, the factors that lead to poor quality research and consequences of poor research reporting may help researchers to self-regulate their research studies from conception.

Key Words: Quality, Research, Publishing,

Clinicians, especially in academic settings, are expected to perform and publish results of their research although this has not always been the case in developing and low resource economies. Most organizations, including funders and grant agencies as well as professional bodies, now mandate that research grants lead to publications in scientific journals. For example, the Medical Council of India has mandated a certain number of publications within a certain period of time as necessary for career advancement within a department.[1-4]  The increased push for research output has led to several unintended consequences including predatory journals, and a rash of publications of questionable quality.[5-7] In this review, we look at certain factors that impact the quality of reported research and explore the potential consequences of research of poor quality.

The research question is of prime importance. Research questions, in clinical fields, have to be relevant to patient care and an improved understanding of patient care domains including reaffirmation of existing knowledge. A practising clinician does not really have to exert to find out relevant research questions. Meticulous care and documentation of practice lends itself to the discovery of several interesting clinical observations that can be translated to research questions.  Given the diverse and often contradictory nature of information in medicine, every question may be of relevance especially when they are well thought out. Additionally, clinical conditions are not simplistic or isolated in nature, evolve over time, and thus lend itself to the possibility of several previously unknown directions being detected. These may include an exploration of social, psychological, environmental, cultural and other factors influencing disease models and are not restricted to isolated models of disease development and management.  We argue that clinicians who are thoughtful about their practice and the clinical care they provide will generate more research questions than they can possibly do.

The case report is an excellent example of clinical observations translating into shared knowledge. Case reports provide the first observation of an interesting finding or variation and form the foundation for well-designed epidemiological studies. Of late, case reports are not accepted by all journals due to a perceived poor quality of reports. [8-10] The poor quality is often associated with inadequate work up of a case and over reaching conclusions or generalizations beyond the scope of the individual case. The adequacy of work up of a case and consequent quality is to be debated especially when one considers cases from low resource settings compared to state of the art institutes.

Research questions need not always be exotic or absolutely new. They can look to re-test previously described studies and to replicate and validate results. Replication of studies is a bedrock in health care research. It is not prudent to rely only on the results of a single or a few studies, however well they may have been done, given the interaction of the person, genes and the environment. Repeated studies of high quality lend themselves to a high quality systematic review and meta- analysis. Needless to say, the quality of systematic reviews and meta-analysis will be poor if the primary research is itself poorly done.

Bias, whether known or unknown, is a factor that may impact on the research. Bias is different from chance variability, which may cause some findings to present differently even though the study design, data collection, analysis and interpretation is well thought out. Bias may occur as a combination of one or more of study design, data collection, analysis and interpretation of results. Bias tends to produce research findings that are not necessarily true even to the extent of being artificially manufactured. Bias may occur in the choice of a study design to answer the question, in the selection of research subjects, in the selection of study variables, in the statistical analysis and even in the interpretation and writing of the research. Bias can include the choice of inappropriate statistical tests to lead to a preferred results, data manipulation through omission of certain variables or data, or even manufacturing of data, selective or distorted reporting focused only on the findings that augment a preferred argument, or through the inefficient use of data, or even conflicts of interest tending to overlook significant findings.[11] Sometimes, a bias may be introduced unknowingly through the failure to notice statistical or clinically significant relations or large measurement errors getting drowned out in noise.[12]

The numbers that you study are important. The estimation of sample size is dependent on the study design and question and involves the input of several factors. Smaller sample sizes mean less power for the study and reduce the positive predictive value of the research finding. Research findings are more likely true when sample sizes are estimated prior to the study and are appropriate to answer the particular research question than with the use of several small studies that maybe later summarized as a meta-analysis.  Research findings are more likely true when effect sizes are large rather than when they are small. Smaller effect sizes may lead to false positive claims of utility where none actually exist. [13-15] This is also true of other measures of association including relative risks and odds ratios.

The study design is important. Each research question has one preferred study design and maybe one alternate study design, with certain limitations, if the question and outcomes are well defined. There has to be valid reasons to choose an alternate design and the interpretation of results have to factor in the limitations of the design. Often, clarity on the study question is lost because of an overlap of several questions or a mix of several related questions that require different designs. In such instances, it is better to restrict the question to the one of highest priority than to mix several questions and lead to chaos. Flexibility in definitions, measures, in outcomes and statistical tests or analytic models can also lead to imprecise results. The field of statistics is dynamic and evolving and there is the potential to choose tests to report only the best results.

The elephant in the room is financial and other conflicts of interest.[16,17] Research findings are less likely to be true when there are large financial interests or conflicts of interest in play.  The reason for research is also a factor to be considered especially with predatory journals and publishing. Poor quality research reports may also arise from poor quality peer review and editorial process, or a biased process that may suppress research with results contrary to expected findings. The lack of reporting of negative findings is a real issue. The peer review and editorial process has to focus on the science and the impact of the research on improved clinical care. Double blind review processes help to reduce bias at the journal level.

Poor research reports have several consequences. The research report may be used to inform clinical practice and become mainstream practice to the detriment of patients. Poor quality research reports maybe used in systematic reviews and meta-analysis driving down the quality of the evidence. Poor quality reports may become teaching and training material with implications on practice and care for generations as they may get into the mainstream as acceptable evidence. For a researcher, having a poor research report published provides an incentive not to aim for high quality work or to improve critical thinking and analytical presentation. One has to consider the large scale effects if governments and society rely on poor research to inform policy decisions.

How can one improve the quality of research and research reporting? Research operates from the principle of uncertainty and it is not possible to know with 100% certainty whether a reported result is true or not. However, several steps can be taken to reduce or minimize the risk of false results. These include drafting well defined and clear research questions, choosing the most appropriate design or an appropriate alternate design based on an understanding of the limitations of the alternate design and its implication on the results.  Estimate an appropriate sample size before the study starts. Choose the study variables and their measurements and definitions carefully and in alignment with the research question. Choose appropriate, well defined and measurable study outcomes. Look at ways to reduce potential bias before the study starts. Draw up a plan of analysis before the study starts, choose a list of appropriate statistical tests based on potential possibilities of data, and explore the known strengths of anticipated relations between data variables. Rather than sticking to a predetermined notion and measure of statistical significance, clinical research is better served by integrating clinical significance and improving understanding of the pre-study odds that determine if a research study is necessary. Statistical significance has to be interpreted in the larger clinical context, in the relevant field at large and under different testing conditions. Almost all of this requires a thorough review of the literature before one starts the study. You have to look at the research study the same way you approach a patient you treat- with respect and dedication to do your best.

The onus on improving the integrity of research has to start and, to a large extent, reside with the individual researcher and research team. This may require a paradigm shift of research from a career, financial, reputation perspective to a more idealistic clinical care perspective. This will also require a redrafting of current medical curriculum integrating research methods and research reporting methods into routine clinical care teaching and evaluation rather than remaining an often neglected chapter in some dust laden book.  Clinical care teaching has to transform into one of collective enquiry rather than absorption of handed down nuggets, and constant questioning (with an improved understanding of answers) to replace hierarchical models of “experience”. Platforms that encourage young investigators and learners to perform and report on research findings, with the experienced seniors moving to a role of active mentorship, have to be developed. Relying on a peer review process and editorial process to filter out poor research is impractical. Peers reviewers and editors are often voluntary and work within competing priorities. It may not always be possible for them to identify and filter out poor research before publication, especially when they deal with multidisciplinary studies or studies where complex statistical models are used.

The quality of research communications can only improve when the quality of research itself improves, which requires a shift in the mind set about the real purpose of research.  Clinical fields may have to develop an acceptable clinical impact score for clinical research based on potential impact to improve care rather than rely on current standards like number of publications, impact factors, h10, i10 and G index.  

 
References:

1. Medical Council of India. Minimum Qualifications for Teachers in Medical Institutions Regulations, 1998 (amended up to 8th June, 2017). Accessed online at http://www.mciindia.org/Rules-and-Regulation/Teachers-Eligibility-Qualifications-Regulations-1998.pdf on June 6, 2018
2. Bandewar SV, Aggarwal A, Kumar R, Aggarwal R, Shani P, Pai SA. Medical Council of India’s amended qualifications for Indian Medical Teachers: Well intended yet half-heearted.  Indian J Urol 2018;34:3-6
3. Sukhlecha A. Research Publications: Should they be mandatory for promotions of medical teachers? J Pharmacol Pharmacother 2011;2: 221-4
4. Ashokan N, Shaji KS. Methods to enhance capacity of medical teachers for research publications. Indian J Public Health 2016;60: 154-8
5. Jain NC. Predatory Journals. Indian J Med Microbiol 2015;33: 426
6. Christopher MM, Young  KM. Awareness of Predatory Open-Access journals among Prospective Veterinary and Medical Authors Attending Scientific Writing Workshops. Front Vet Sci 2015;2:22
7. Dadkhah M, Bianciardi G. Ranking Predatory Journals: Solve the problem instead of removing it. Adv Pharm Bull 2016;6: 1-4
8. Talat B, Mayers A, Baldwin DS. Quality of case reports of adverse drug reactions with psychotropic drugs: a 25 year review. Human Psychopharmacol 2013;28: 413
9. Davenport TE. Clinical case reporting in the peer-reviewed physical therapy literature: Time to move towards functioning. Physiother Res Int 2015;20::220-30
10. Stevens DP. The context is the news in healthcare improvement case reports. Qual Saf Health Care 2010;19:62-3
11. Topol EJ. Failing the public health-Rofecoxib, Merck and the FDA. N Engl J Med 2014;351:1707-09
12. Kelsey JL, Whittemore AS, Evans AS, Thompson WD. Methods in observational epidemiology (2nd ed) 1996. New York: Oxford U Press:p432
13. Ioannidis JP. Genetic Associations: False or True? Trends Mol Med 2003;9:135-38
14. Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med 2000;19:453-473
15. Taubes G. Epidemiology faces its limits. Science 1995;269:164-69
16. Papanilolaou GN, Baltogianni MS,Contopoulos-Ioannidis DG, Haidich AB, Giannakakis IA, et al. Reporting of conflicts of interest in guidelines of preventive and therapeutic interventions. BMC Med Res Methodol 2001;1:3
17. Krimsky S, Rothernberg LS, Stott P, Kyle G. Scientific Journals and their authors’ financial interests: A pilot study. PsyhcotherPsychosom 1998;67:194-201