Time to be positive about negative data?

  • T. Vincent
    Correspondence
    Address correspondence and reprint requests to: T. Vincent, Arthritis Research UK Centre for OA Pathogenesis, Kennedy Institute of Rheumatology, University of Oxford, Roosevelt Drive, Oxford OX3 7FY, UK
    Affiliations
    Arthritis Research UK Centre for OA Pathogenesis, Kennedy Institute of Rheumatology, University of Oxford, Roosevelt Drive, Oxford OX3 7FY, UK
    Search for articles by this author
  • A.-M. Malfait
    Correspondence
    Address correspondence and reprint requests to: A.-M. Malfait, Department of Internal Medicine, Division of Rheumatology & Department of Biochemistry, Rush University Medical Center, 1611 W. Harrison St, Suite 510, Chicago, IL 60612, USA.
    Affiliations
    Department of Internal Medicine, Division of Rheumatology & Department of Biochemistry, Rush University Medical Center, 1611 W. Harrison St, Suite 510, Chicago, IL 60612, USA
    Search for articles by this author
      With no positivity, there is no hope; with no negativity, there is no improvement.– Criss Jami, Healology
      Karl Popper (1902–1994), one of the greatest philosophers of science of the 20th century, writes in Conjectures and Refutations (Harper & Row, 1963): “Criticism of our conjectures is of decisive importance: by bringing out our mistakes it makes us understand the difficulties of the problem which we are trying to solve. This is how we become better acquainted with our problem, and able to propose more mature solutions: the very refutation of a theory – that is, of any serious tentative solution to our problem-is always a step forward that takes us nearer to the truth. And this is how we can learn from our mistakes”.
      This special issue of Osteoarthritis and Cartilage was devised to capture important negative findings in in vivo models of osteoarthritis (OA), both with respect to pathogenesis and treatment of the disease. Negative results are mostly met with disappointment and dismissed by researchers as ‘failed’ experiments. It is somewhat ironic – in a disease where we collectively lack mechanistic understanding and we have no benchmarks for what is right (true) and what is wrong (false)- that we seem to deem negative studies (i.e., studies where the results force us to refute our hypothesis) as being less informative than positive ones. This is concerning when we consider that the use of murine models in OA research has expanded exponentially since the turn of this century (see Fig. 1), and they are increasingly being used as a screening tool for studying putative mediators of disease, often without strong mechanistic rationale.
      Fig. 1
      Fig. 1Number of publications by year using mice in OA research. Data was generated by I. von Loga and T. Vincent using the search terms ((((mouse) OR murine) OR mice)) AND osteoarthritis in Pubmed.
      Many of our colleagues have unpublished negative results hidden away in drawers, experiments in knockout mice that had no clear phenotype being a common example. These findings are sometimes presented in abstract format, but often never formally submitted. The responsibility of failure to publish negative findings stems in part from publishers, granting agencies and Universities who want to see positive data being delivered, and in part from the researchers themselves. Scientists may have a higher threshold for submitting negative results, since positive results are just more exciting than negative ones, and we all shy away from the demanding task of preparing a full-length manuscript that only serves to communicate “failed” experiments, owning up to the fact that we were mistaken. Even as reviewers, we may be more critical when evaluating negative studies, paying extra attention to whether the study is adequately powered and all the proper controls have been included.
      The response to the call for the current Special Themed Issue on Negative In Vivo Studies was not overwhelming. We received 29 original research submissions, seven of which described findings in knock-out mice, nine submissions that tested therapies or interventions, 11 that evaluated repair strategies, and two others. Ultimately, just seven original submissions were accepted, which amounts to a 25% acceptance rate, in line with the usual acceptance rate of this Journal. We have selected a number of studies where there was a clear and compelling rationale for performing the study, where the study was robustly designed and where it tells us something important about OA pathogenesis. We have also collated four expert reviews on aspects of in vivo OA studies in order to improve quality, design and reporting of studies to gain optimal impact from our scientific endeavors.
      Positive bias in scientific reporting is dangerous as, through citation and systematic review, it leads to an overestimate of the importance of a given pathway or treatment in disease. This contributes to failed clinical studies and the massive attrition that the pharmaceutical industry suffers during development of novel therapies. There is currently a big push toward reducing the positive bias in the scientific literature at large, also at the level of publishers, such as Elsevier https://www.elsevier.com/authors-update/story/innovation-in-publishing/why-science-needs-to-publish-negative-results. There are several new online journals where negative or inconclusive data can find a home (Journal of Negative Results in Biomedicine, The All Results Journals, Journal of Articles in Support of the Null Hypothesis, and more), and new ways to provide access to negative data have emerged (e.g., https://pubpeer.com/). Negative findings are also increasingly represented in broad-scope journals such as Disease Models & Mechanisms and PLoS ONE (the latter recently published a virtual collection, entitled “The Missing Pieces: A Collection of Negative, Null and Inconclusive Results”). Even so, new journals launched with the specific scope of publishing negative findings often do not attract as many papers, demonstrating that it is the underlying scientific culture that requires change and not only journal policies.
      The problem is not simply in the reporting but in the quality and design of the study, as argued in the review by Smith et al. in this issue
      • Smith M.M.
      • Clarke E.C.
      • Little C.B.
      Considerations for the design and execution of protocols for animal research and treatment to improve reproducibility and standardization: “DEPART well-prepared and ARRIVE safely”.
      . A false negative result is as unhelpful as a false positive result and much research funding is wasted on poorly designed studies. There are certain challenges to how we define a negative result. This cannot simply be defined by an insignificant P-value, as it will depend wholly on the size of effect the study has been powered for
      • Ranstam J.
      • Cook J.A.
      Considerations for the design, analysis and presentation of in vivo studies.
      . Our clinical colleagues can help us here: they would define a negative study as one in which the primary endpoint of the study has not been met. In other words, they decide at the outset of the study what they are going to measure, what effect size they would regard as clinically significant (and therefore relevant) and the numbers required to power the study appropriately
      • Ranstam J.
      • Cook J.A.
      Considerations for the design, analysis and presentation of in vivo studies.
      . This is rarely done in pre-clinical modeling but should be encouraged. We then have the additional problem of deciding what a clinically relevant change in cartilage degradation score is in a mouse. Most published positive studies tend to show an effect size of around 40–60%; therefore, does this mean that any effect size less than this negative?
      It wouldn't be truthful to say that Rheumatology journals do not publish negative studies. There are several notable negative in vivo OA studies that have strongly impacted the field. For instance, Adamts4 null mice were found not to be protected from experimental OA
      • Glasson S.S.
      • Askew R.
      • Sheppard B.
      • Carito B.A.
      • Blanchet T.
      • Ma H.L.
      • et al.
      Characterization of and osteoarthritis susceptibility in ADAMTS-4-knockout mice.
      and Adamts4/Adamts5 double knockout animals showed no demonstrable increased protection over Adamts5 null mice following surgical destabilization
      • Majumdar M.K.
      • Askew R.
      • Schelling S.
      • Stedman N.
      • Blanchet T.
      • Hopkins B.
      • et al.
      Double-knockout of ADAMTS-4 and ADAMTS-5 in mice results in physiologically normal animals and prevents the progression of osteoarthritis.
      . Similarly, ablation of Adamts1 offered no protection from accelerated aggrecan degradation in an inflammatory model of arthritis
      • Little C.B.
      • Mittaz L.
      • Belluoccio D.
      • Rogerson F.M.
      • Campbell I.K.
      • Meeker C.T.
      • et al.
      ADAMTS-1-knockout mice do not exhibit abnormalities in aggrecan turnover in vitro or in vivo.
      . In combination with the clear protection observed in Adamts5 null mice, both in surgical OA and in a model of inflammatory arthritis
      • Glasson S.S.
      • Askew R.
      • Sheppard B.
      • Carito B.
      • Blanchet T.
      • Ma H.L.
      • et al.
      Deletion of active ADAMTS5 prevents cartilage degradation in a murine model of osteoarthritis.
      • Stanton H.
      • Rogerson F.M.
      • East C.J.
      • Golub S.B.
      • Lawlor K.E.
      • Meeker C.T.
      • et al.
      ADAMTS5 is the major aggrecanase in mouse cartilage in vivo and in vitro.
      , these findings strongly skewed efforts to block aggrecanase activity toward ADAMTS-5 as the prime target, efforts that have unfortunately not yet been translated into the clinic
      • Fosang A.J.
      ADAMTS-5 takes centre stage in new developments for aggrecanase inhibitors.
      .
      Published negative studies are often used to refute scientific prejudice and in the OA field this has mainly centered on the role of inflammatory molecules. Surprisingly, mice in which either IL1β, interleukin 1 converting enzyme (ICE), MMP3, or iNOS are deleted have slightly higher disease scores rather than protection early after partial meniscectomy
      • Clements K.M.
      • Price J.S.
      • Chambers M.G.
      • Visco D.M.
      • Poole A.R.
      • Mason R.M.
      Gene deletion of either interleukin-1beta, interleukin-1beta-converting enzyme, inducible nitric oxide synthase, or stromelysin 1 accelerates the development of knee osteoarthritis in mice after surgical transection of the medial collateral ligament and partial medial meniscectomy.
      . Lack of chondroprotection following surgical joint destabilization after deletion of the genes encoding the cyclooxygenase (COX) enzymes, COX-1 or COX-2, was reported in a Brief Report
      • Fukai A.
      • Kamekura S.
      • Chikazu D.
      • Nakagawa T.
      • Hirata M.
      • Saito T.
      • et al.
      Lack of a chondroprotective effect of cyclooxygenase 2 inhibition in a surgically induced model of osteoarthritis in mice.
      . This valuable paper also carried results of two other important negative studies: lack of chondroprotection in IL1α/IL1β double knockout and in TNFα knockout strains. These results were reported only in the discussion with data available through the corresponding author, and thus are infrequently cited. Conversely, in a review by Glasson, modest protection in the IL1β knockout mouse following surgical destabilization was reported
      • Glasson S.S.
      In vivo osteoarthritis target validation utilizing genetically-modified mice.
      . Critical evaluation of these results is challenging especially as the latter paper was in a poorly accessible journal. In the current Special Issue, van Dalen and colleagues revisit IL-1 as a target in OA. Despite performing these studies in arguably a more inflammatory model of OA (collagenase induced), they show that IL-1α/β deletion confers no reduction in cartilage loss or synovial inflammation
      • van Dalen S.C.
      • Blom A.B.
      • Sloetjes A.W.
      • Helsen M.M.
      • Roth J.
      • Vogl T.
      • et al.
      Interleukin-1 is not involved in synovial inflammation and cartilage destruction in collagenase-induced osteoarthritis.
      . In fact, the data suggest a trend towards increased cartilage damage in the knockout animals compared with wild type, agreeing with increased disease reported by Clements et al. in a surgical model
      • Clements K.M.
      • Price J.S.
      • Chambers M.G.
      • Visco D.M.
      • Poole A.R.
      • Mason R.M.
      Gene deletion of either interleukin-1beta, interleukin-1beta-converting enzyme, inducible nitric oxide synthase, or stromelysin 1 accelerates the development of knee osteoarthritis in mice after surgical transection of the medial collateral ligament and partial medial meniscectomy.
      . On balance, despite the potent catabolic effects of IL1 on cartilage in vitro, IL-1 continues likely to prove a disappointing target in OA.What gets us into trouble is not what we don't know, it's what we know for sure that just ain't so.”– Mark Twain
      We also need to be cautious about how we interpret in vivo models when using genetic deletion. Constitutive deletion of a gene is the most common gene-modifying tool used in mouse studies, but there are limitations associated with potential mild undetected chondrodysplasia that may impact OA severity, or compensatory mechanisms that have evolved over time. Conditional deletion of the gene of interest postnatally has the advantage of avoiding a developmental phenotype and can be temporally controlled. In this issue, two papers complemented knockout studies with overexpression of the gene of interest by adenoviral carrier
      • Choi W.S.
      • Chun J.S.
      Upregulation of lipocalin-2 (LCN2) in osteoarthritic cartilage is not necessary for cartilage destruction in mice.
      • Kim H.E.
      • Rhee J.
      • Park S.
      • Yang J.
      • Chun J.S.
      Upregulation of Atrogin-1/FBXO32 is not necessary for cartilage destruction in mouse models of osteoarthritis.
      . Both strategies failed to show a disease modifying effect following surgical joint destabilization. Timing of genetic deletion or treatment may be critical for disease outcome. In this issue, we report the failure to show a significant reduction in cartilage degradation in Ccl2 and Ccr2 constitutive knockout mice
      • Miotla Zarebska J.
      • Chanalaris A.
      • Driscoll C.
      • Burleigh A.
      • Miller R.E.
      • Malfait A.M.
      • et al.
      CCL2 and CCR2 regulate pain-related behaviour and early gene expression in post-traumatic murine osteoarthritis but contribute little to chondropathy.
      , yet targeting the pathway after the initial injury appears to be effective at inhibiting disease
      • Longobardi L.
      • Temple J.D.
      • Tagliafierro L.
      • Willcockson H.
      • Esposito A.
      • D'Onofrio N.
      • et al.
      Role of the C-C chemokine receptor-2 in a murine model of injury-induced osteoarthritis.
      . In other words, molecules may have different roles at different times of disease.
      There are other considerations to publishing negative data, notably ethical ones. Is it right to do an experiment and then not share knowledge from it? If an experiment was worth doing in the first place, then do we not have a duty to publish so that other groups do not end up repeating unnecessarily? Withholding data in this way goes against our desire to conform to the ‘3Rs’ (reduction, refinement and replacement) in animal research and wastes valuable research resources, especially in our current climate where research funding is ever declining.
      So what are the solutions? One simple solution would be to encourage presentation of more negative data at international meetings in abstract form; lobbying research societies to rate these as they would a good negative clinical study. We could welcome the publication of more negative data in supplementary figures of relevant papers. A more radical option would be to create an online repository (with some editorial control) where groups can deposit negative in vivo data with detailed experimental methodology, but without having to prepare a whole manuscript.
      Ultimately, we need to change the scientific ethos in our research institutions. Imagine the following scenario:
      Student: “…but what happens if my hypothesis is wrong and the study does not show a positive result?”

      How should the supervisor answer?

      • A.
        “That will be unfortunate, your chances of getting a good postdoc position will be diminished; that's the luck of the science lottery!” or
      • B.
        “Don't worry; it's always possible to squeeze some positive result out of a negative study!” or
      • C.
        “A negative result in a well designed and thoughtful study is always valuable!”.

      Contributions

      Both authors wrote the manuscript and approved the final version.

      Conflict statement

      Dr. Vincent is on the Editorial Board of OAC, and Dr. Malfait serves as an Associate Editor.

      References

        • Smith M.M.
        • Clarke E.C.
        • Little C.B.
        Considerations for the design and execution of protocols for animal research and treatment to improve reproducibility and standardization: “DEPART well-prepared and ARRIVE safely”.
        Osteoarthritis Cartilage. 2017; 25: 354-363
        • Ranstam J.
        • Cook J.A.
        Considerations for the design, analysis and presentation of in vivo studies.
        Osteoarthritis Cartilage. 2017; 25: 364-368
        • Glasson S.S.
        • Askew R.
        • Sheppard B.
        • Carito B.A.
        • Blanchet T.
        • Ma H.L.
        • et al.
        Characterization of and osteoarthritis susceptibility in ADAMTS-4-knockout mice.
        Arthritis Rheum. 2004; 50: 2547-2558
        • Majumdar M.K.
        • Askew R.
        • Schelling S.
        • Stedman N.
        • Blanchet T.
        • Hopkins B.
        • et al.
        Double-knockout of ADAMTS-4 and ADAMTS-5 in mice results in physiologically normal animals and prevents the progression of osteoarthritis.
        Arthritis Rheum. 2007; 56: 3670-3674
        • Little C.B.
        • Mittaz L.
        • Belluoccio D.
        • Rogerson F.M.
        • Campbell I.K.
        • Meeker C.T.
        • et al.
        ADAMTS-1-knockout mice do not exhibit abnormalities in aggrecan turnover in vitro or in vivo.
        Arthritis Rheum. 2005; 52: 1461-1472
        • Glasson S.S.
        • Askew R.
        • Sheppard B.
        • Carito B.
        • Blanchet T.
        • Ma H.L.
        • et al.
        Deletion of active ADAMTS5 prevents cartilage degradation in a murine model of osteoarthritis.
        Nature. 2005; 434: 644-648
        • Stanton H.
        • Rogerson F.M.
        • East C.J.
        • Golub S.B.
        • Lawlor K.E.
        • Meeker C.T.
        • et al.
        ADAMTS5 is the major aggrecanase in mouse cartilage in vivo and in vitro.
        Nature. 2005; 434: 648-652
        • Fosang A.J.
        ADAMTS-5 takes centre stage in new developments for aggrecanase inhibitors.
        Osteoarthritis Cartilage. 2015; 23: 1231-1232
        • Clements K.M.
        • Price J.S.
        • Chambers M.G.
        • Visco D.M.
        • Poole A.R.
        • Mason R.M.
        Gene deletion of either interleukin-1beta, interleukin-1beta-converting enzyme, inducible nitric oxide synthase, or stromelysin 1 accelerates the development of knee osteoarthritis in mice after surgical transection of the medial collateral ligament and partial medial meniscectomy.
        Arthritis Rheum. 2003; 48: 3452-3463
        • Fukai A.
        • Kamekura S.
        • Chikazu D.
        • Nakagawa T.
        • Hirata M.
        • Saito T.
        • et al.
        Lack of a chondroprotective effect of cyclooxygenase 2 inhibition in a surgically induced model of osteoarthritis in mice.
        Arthritis Rheum. 2012; 64: 198-203
        • Glasson S.S.
        In vivo osteoarthritis target validation utilizing genetically-modified mice.
        Curr Drug Targets. 2007; 8: 367-376
        • van Dalen S.C.
        • Blom A.B.
        • Sloetjes A.W.
        • Helsen M.M.
        • Roth J.
        • Vogl T.
        • et al.
        Interleukin-1 is not involved in synovial inflammation and cartilage destruction in collagenase-induced osteoarthritis.
        Osteoarthritis Cartilage. 2017; 25: 385-396
        • Choi W.S.
        • Chun J.S.
        Upregulation of lipocalin-2 (LCN2) in osteoarthritic cartilage is not necessary for cartilage destruction in mice.
        Osteoarthritis Cartilage. 2017; 25: 401-405
        • Kim H.E.
        • Rhee J.
        • Park S.
        • Yang J.
        • Chun J.S.
        Upregulation of Atrogin-1/FBXO32 is not necessary for cartilage destruction in mouse models of osteoarthritis.
        Osteoarthritis Cartilage. 2017; 25: 397-400
        • Miotla Zarebska J.
        • Chanalaris A.
        • Driscoll C.
        • Burleigh A.
        • Miller R.E.
        • Malfait A.M.
        • et al.
        CCL2 and CCR2 regulate pain-related behaviour and early gene expression in post-traumatic murine osteoarthritis but contribute little to chondropathy.
        Osteoarthritis Cartilage. 2017; 25: 406-412
        • Longobardi L.
        • Temple J.D.
        • Tagliafierro L.
        • Willcockson H.
        • Esposito A.
        • D'Onofrio N.
        • et al.
        Role of the C-C chemokine receptor-2 in a murine model of injury-induced osteoarthritis.
        Osteoarthritis Cartilage. 2016; https://doi.org/10.1016/j.joca.2016.11.004