PreprintPDF AvailableAssessing Confidence in the Results of Network Meta-Analysis (Cinema)April 2019DOI:10.1101/597047Authors: Adriani NikolakopoulouUniversity of Ioannina Julian PT HigginsJulian PT HigginsThis person is not on ResearchGate, or hasn t claimed this research yet. Thodoris PapakonstantinouUniversity Medical Center Freiburg Anna ChaimaniFrench Institute of Health and Medical ResearchShow all 7 authorsHide Download file PDFDownload file PDFRead filePreprints and early-stage research may not have been peer reviewed yet.Download file PDFDownload file PDFRead fileDownload citation Copy link Link copied Read file Download citation Copy link Link copiedCitations (22)References (48)AbstractEvaluation of the credibility of results from a meta-analysis has become an intrinsic part of the evidence synthesis process. We present a methodological framework to evaluate Confidence In the results from Network Meta-Analysis (CINeMA) when multiple interventions are compared. CINeMA considers six domains and we outline the methods used to form judgements about within-study bias, across-studies bias, indirectness, imprecision, heterogeneity and incoherence. Key to judgements about within-study bias and indirectness is the percentage contribution matrix, which shows how much information each study contributes to the results from network meta-analysis. The use of contribution matrix allows the semi-automation of the process, implemented in a freely available web application (cinema.ispm.ch). In evaluating imprecision, heterogeneity and inconsistency we consider the impact of these components of variability in forming clinical decisions. Via three examples, we show that CINeMA improves transparency and avoids the selective use of evidence when forming judgements, thus limiting subjectivity in the process. CINeMA is easy to apply even in large and complicated networks, like a network involving 18 different antidepressant drugs. Discover the world s research20+ million members135+ million publications700k+ research projectsJoin for freePublic Full-texts 2597047.full.pdfContent available from CC BY-NC-ND 4.0:597047.full.pdfAssessing Confidence in the Results of Network Meta-Analysis (Cinema).pdfContent uploaded by Thodoris PapakonstantinouAuthor contentAll content in this area was uploaded by Thodoris Papakonstantinou on Apr 08, 2019 Content may be subject to copyright.Assessing Confidence in the Results of Network Meta-Analysis (Cinema).pdfContent available from CC BY-NC-ND 4.0:597047.full.pdfAssessing Confidence in the Results of Network Meta-Analysis (Cinema).pdfAvailable via license: CC BY-NC-ND 4.0Content may be subject to copyright. 1 ASSESSING CONFIDENCE IN THE RESULTS OF NETWORK META-1 ANALYSIS (CINEMA) 2 3 Adriani Nikolakopoulou1 , Julian PT Higgins2, Theodore Papakonstantinou1, Anna 4 Chaimani3,4,5, Cinzia Del Giovane6, Matthias Egger1, Georgia Salanti1. 5 6 7 1 Institute of Social and Preventive Medicine, University of Bern, Switzerland 8 2 Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, U.K. 9 3 School of Medicine, Paris Descartes University, Paris, France. 10 4 INSERM, UMR1153 Epidemiology and Statistics, Sorbonne Paris Cité Research 11 Center, Paris, France. 12 5 French Cochrane Center, Hôpital Hôtel-Dieu, Paris, France. 13 6 Institute of Primary Health Care (BIHAM), University of Bern, Bern, Switzerland. 14 15 16 6138 words 17 5 tables, 4 figures, 2 boxes 18 19 20 21 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 2 Abstract 22 23 Evaluation of the credibility of results from a meta-analysis has become an intrinsic 24 part of the evidence synthesis process. We present a methodological framework to evaluate 25 Confidence In the results from Network Meta-Analysis (CINeMA) when multiple 26 interventions are compared. CINeMA considers six domains and we outline the methods 27 used to form judgements about within-study bias, across-studies bias, indirectness, 28 imprecision, heterogeneity and incoherence. Key to judgements about within-study bias and 29 indirectness is the percentage contribution matrix, which shows how much information 30 each study contributes to the results from network meta-analysis. The use of contribution 31 matrix allows the semi-automation of the process, implemented in a freely available web 32 application (cinema.ispm.ch). In evaluating imprecision, heterogeneity and inconsistency we 33 consider the impact of these components of variability in forming clinical decisions. Via 34 three examples, we show that CINeMA improves transparency and avoids the selective use 35 of evidence when forming judgements, thus limiting subjectivity in the process. CINeMA is 36 easy to apply even in large and complicated networks, like a network involving 18 different 37 antidepressant drugs. 38 39 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 3 INTRODUCTION 40 Network meta-analysis has become an increasingly popular tool for developing 41 treatment guidelines and making recommendations on reimbursement. However, less than 42 one per cent of published network meta-analyses assess the credibility of their conclusions 43 (1). The Grading of Recommendations Assessment, Development and Evaluation (GRADE) 44 approach requires such an assessment of the confidence in the results from systematic 45 reviews and meta-analyses, and many organizations, including the World Health 46 Organization (WHO), have adopted the GRADE approach (2,3). Based on GRADE, two 47 systems have been proposed to evaluate the credibility of results from network meta-48 analyses (4,5). However, the complexity of the methods and lack of suitable software have 49 limited their uptake. 50 In this article we introduce the methodology underpinning the CINeMA approach 51 (Confidence In Network Meta-Analysis), and present the advances that have recently been 52 implemented in a freely available web application (cinema.ispm.ch) (6). CINeMA is based on 53 the GRADE framework, with several conceptual and semantic differences (5). It covers six 54 confidence domains: within-study bias (referring to the impact of risk of bias in the included 55 studies), across-studies bias (referring to publication and other reporting bias), indirectness, 56 imprecision, heterogeneity and incoherence. CINeMA assigns judgements at three levels (no 57 concerns, some concerns or major concerns) to each of the six domains. Judgements across 58 the six domains are then summarized to obtain four levels of confidence for each relative 59 treatment effect, corresponding to the usual GRADE approach: very low, low, moderate or 60 high. 61 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 4 Most network meta-analyses include only randomized controlled trials (RCTs), so we 62 will focus on this study design, and on relative treatment effects. A network meta-analysis 63 involves the integration of direct and indirect evidence in a network of relevant trials. We 64 assume that evaluation of the credibility of results takes place once all primary analyses and 65 sensitivity analyses have been undertaken. We assume that reviewers have implemented 66 their pre-specified study inclusion criteria, which may include risk of bias considerations, 67 and have obtained the best possible estimates of relative treatment effects using 68 appropriate statistical methods (e.g. those described in (7–10)). The question is then how to 69 make judgements about the credibility of relative treatment effects, given that trials with 70 variable risk of bias, precision, relevance and heterogeneity contribute information to the 71 estimate. 72 This paper addresses how judgements should be formed about the six CINeMA 73 domains. We illustrate the methods using three examples: a network of trials that compare 74 outcomes of various diagnostic strategies in patients with suspected acute coronary 75 syndrome (11), a network of trials comparing the effectiveness of 18 antidepressants for 76 major depression (12), and a network comparing adverse events of statins (13). The three 77 examples are introduced in Error! Reference source not found.. All analyses were done in R 78 software using the netmeta package and the CINeMA web application (Box 2) (6,14). 79 WITHIN-STUDY BIAS 80 BACKGROUND AND DEFINITIONS 81 Within-study bias refers to shortcomings in the design or conduct of a study that can 82 lead to an estimated relative treatment effect that systematically differs from the truth. In 83 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 5 our framework we assume that studies have been assessed for risk of bias. The majority of 84 published systematic reviews of RCTs currently use a tool developed by Cochrane to 85 evaluate risk of bias (15). This tool classifies studies as having low, unclear or high risk of 86 bias for various bias components (such as allocation concealment, attrition, blinding etc.), 87 and these judgements are then summarized across domains. A revision of the tool takes a 88 similar approach but labels the levels as low risk of bias, some concerns and high risk of bias 89 (16). 90 THE CINEMA APPROACH 91 While it is straightforward to gauge the impact of within-study biases on the summary 92 relative treatment effect in a pairwise meta-analysis (17), in network meta-analysis studies 93 contribute data to the estimation of each summary effect in a complex manner. In the first 94 example discussed below we show the complexity underpinning the flow of information in 95 the network of diagnostic modalities used to detect coronary artery disease. A treatment 96 comparison directly evaluated in studies with low risk of bias might also be estimated 97 indirectly (via a common comparator) using studies at high risk of bias, and vice versa. While 98 studies at low risk of bias are expected to provide more credible results, it is often 99 impractical to restrict the analysis to such studies. The treatment comparison of interest 100 might not have been tested directly in any trial, or tested in only a few small trials with high 101 risk of bias. Thus, even when direct evidence is present, judgements about the relative 102 treatment effect cannot ignore the risk of bias in the studies providing indirect evidence. 103 If direct evidence is supplemented by indirect evidence via exactly one intermediate 104 comparator, the risk of bias in such a one-step loop is considered along with the direct 105 evidence. In complex networks, indirect evidence is often obtained via several routes, 106 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 6 including one-step loops and loops involving several steps (see example). In general, it is not 107 desirable to derive judgements by considering only the risk of bias in studies in a single one-108 step loop (4,18). This is because most studies in a network contribute some indirect 109 information to every estimate of a relative treatment effect. Studies contribute more when 110 their results are precise (e.g. large studies), when they provide direct evidence or when the 111 indirect evidence does not involve many “steps”. For example, studies in a one-step indirect 112 comparison contribute more than studies of the same precision in a two-step indirect 113 comparison. We can quantify the contribution made by each study to each relative 114 treatment effect on a 0 to 100 percent scale. These quantities can be written as a 115 ‘percentage contribution matrix’, as shown elsewhere (19). 116 CINeMA combines the studies’ contributions with the risk of bias judgements to 117 evaluate study limitation for each estimate of a relative treatment effect from a network 118 meta-analysis. It uses the percentage contribution matrix to approximate the contribution 119 of each study and then stratifies the percentage contribution from studies judged to be at 120 low, moderate and high risk of bias. Using different colors, study limitations in direct 121 comparisons can be shown graphically in the network plot, while study limitations in the 122 estimates from a network meta-analysis are presented for each comparison in bar charts. 123 EXAMPLE: COMPARING DIAGNOSTIC MODALITIES TO DETECT CORONARY ARTERY DISEASE 124 Consider the comparison of Exercise ECG versus Standard care (Box 1). The direct 125 evidence from a single study is at low risk of bias (3-arm study 12); so there are no study 126 limitations when interpreting the direct odds ratio of 0.42 (Table 1). However, the odds ratio 127 0.52 from the network meta-analysis is estimated also by using indirect information via 128 seven studies that compare standard care and CCTA and one study comparing exercise ECG 129 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 7 and CCTA. Additionally, we have indirect evidence via stress echo. The risk of bias in these 130 eleven studies providing indirect evidence varies. Every study in the two one-step loops 131 contributes information proportional to its precision (the inverse of the squared standard 132 error, largely driven by sample size). Consequently, some judgement about study limitations 133 for the indirect evidence can be made by considering that a there is a large amount of 134 information from studies at high risk of bias (2162 participants randomized) and low risk of 135 bias (2788 participants) and relatively little information from studies at moderate risk of bias 136 (362 participants). Direct evidence from the small study number 12 (130 participants) at low 137 risk of bias is considered separately, as it has greater influence than the indirect evidence. 138 Calculations become more complicated because studies in the indirect comparisons 139 contribute information not only proportional to their study precision but also to their 140 location in the network. Indirect evidence about exercise ECG versus SPECT-MPI comes from 141 two one-step loops (via CCTA or via Standard Care) and three two-step loops (via CCTA-142 Standard Care, Stress Echo-Standard Care, Standard Care-CCTA) (Figure 1A). In each loop of 143 evidence, a different subgroup of studies contributes indirect information and their sizes 144 and risks of bias vary. For the odds ratio from the network meta-analysis comparing exercise 145 ECG and SPECT-MPI, study 2 with sample size 400 will be more influential than study 8 (with 146 sample size 1392) because study 2 contributes one-step indirect evidence (via standard 147 care). 148 Table 2 shows the percentage contribution matrix for the network and the columns 149 represent the studies, grouped by comparison. The rows represent all relative treatment 150 effects from network meta-analysis. The matrix entries show how much each study 151 contributes to the estimation of each relative treatment effect. This information combined 152 with the risk of bias judgements can be presented as a bar chart, as shown in Figure 2. Now, 153 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 8 it is much easier to judge study limitations for each odds ratio; the larger the contribution 154 from studies at high or moderate risk of bias, the more concerned we are about study 155 limitations. Using this graph, we can infer that the total evidence from the network meta-156 analysis for the comparison of exercise ECG with SPECT-MPI involves low, moderate and 157 high risk of bias studies with percentages 44%, 32% and 24%, respectively. 158 The CINeMA software offers the option to automate production of judgments, based 159 on the data presented in these bar graphs combined with specific rules. One possible rule is 160 to compute a weighted average level of risk of bias, assigning scores of −1, 0 and 1 to low, 161 moderate and high risk of bias. For the comparison exercise ECG vs SPECT-MPI, this would 162 produce a weighted score of 0.44 × (-1) + 0.32 × 0 + 0.24 × 1 = −0.20, which corresponds to 163 some concerns in the scoring scheme. 164 EXAMPLE: COMPARING ANTIDEPRESSANTS 165 We will focus on evaluating the results for three comparisons; amitriptyline vs 166 milnacipran (one direct study at low and one at moderate risk of bias), mirtazapine versus 167 paroxetine (three direct studies at low risk of bias and two at moderate) and amitriptyline vs 168 clomipramine (no direct studies). The odds ratios for treatment response are presented in 169 Table 3. We use this example to illustrate the use of sensitivity analysis and how it can 170 inform the amount of contribution of studies at moderate and high risk of bias that we can 171 tolerate. 172 For the first two treatment comparisons in Table 3, the contribution from studies at 173 low risk of bias is more than 50%. Moreover, the sensitivity analysis excluding studies at 174 moderate risk of bias provides results comparable to those obtained from all studies. Thus, 175 one can derive the judgment of no concerns for amitriptyline versus milnacipran and 176 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 9 mirtazapine versus paroxetine. However, the estimation of the relative treatment effect of 177 amitriptyline versus clomipramine comes by more than 60% from studies at moderate risk 178 of bias. Given also that the odds ratio from the sensitivity analysis is quite different from to 179 the one obtained from all studies, we judge as some concerns the amitriptyline versus 180 clomipramine comparison. 181 ACROSS-STUDIES BIAS 182 BACKGROUND AND DEFINITIONS 183 Across-studies bias occurs when the studies included in the systematic review are not 184 a representative sample of the studies undertaken. This phenomenon can be the result of 185 the suppression of statistically significant (or “negative”) findings (publication bias), their 186 delayed publication (time-lag bias) or omission of unfavorable study results (outcome 187 reporting bias). The presence and the impact of such biases has been well documented (20–188 26). Across-studies bias is a missing data problem, and hence it is impossible to conclude 189 with certainty for or against its presence in a given dataset. Consequently, and in agreement 190 with the GRADE system, CINeMA assumes two possible descriptions for across-studies bias: 191 suspected and undetected. 192 THE CINEMA APPROACH 193 Assessment of the risk of across-studies bias follows considerations on pairwise meta-194 analysis (27). Conditions associated with ‘suspected’ across-studies bias include: 195 - Failure to include unpublished data and data from grey literature. 196 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 10 - The meta-analysis is based on a small number of positive early findings, for example for 197 a drug newly introduced on the market (as early evidence is likely to overestimate its 198 efficacy and safety) (27). 199 - The treatment comparison is studied exclusively or primarily in industry-funded trials 200 (28,29). 201 - There is previous evidence documenting the presence of reporting bias. For example the 202 study by Turner et al. documented publication bias in placebo-controlled antidepressant 203 trials (30). 204 Across-studies bias is considered ‘undetected’ when 205 - Data from unpublished studies have been identified and their findings agree with those 206 in published studies 207 - There is a tradition of prospective trial registration in the field and protocols or clinical 208 trial registries do not indicate important discrepancies with published reports 209 - Empirical examination of patterns of results between small and large studies, using the 210 comparison-adjusted funnel plot (31,32), regression models (33) or selection models 211 (34) do not indicate that results from small studies differ from those in published 212 studies. 213 EXAMPLE: COMPARING ANTIDEPRESSANTS 214 The literature search retrieved supplementary and unpublished information from 215 clinical trial registries, regulatory agencies’ repositories and drug companies’ websites 216 (particularly for the newest and most recently marketed antidepressants). Results from 217 published and unpublished studies did not differ materially, no asymmetry was observed in 218 the funnel plot (12) and meta-regression did not indicate an association between study 219 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 11 precision and study odds ratio. However, the authors decided that they cannot completely 220 rule out the possibility that some studies are missing because the field of antidepressant 221 trials has been shown to be prone to publication bias. Consequently, the review team 222 decided to assume that across-studies bias was ‘suspected’ for all drug comparisons. 223 INDIRECTNESS 224 BACKGROUND AND DEFINITIONS 225 Systematic reviews are based on a focused research question, with a clearly defined 226 population, intervention and setting of interest. In the GRADE framework for pairwise meta-227 analysis, indirectness refers to the relevance of the included studies to the research 228 question (35). Study populations, interventions, outcomes and study settings should match 229 the inclusion criteria of the systematic review but might not be representative of the 230 settings, populations or outcomes about which reviewers want to make inferences. For 231 example, a systematic review aiming to provide evidence about treating middle-aged adults 232 might identify studies in elderly patients; these studies will have an indirect relevance. 233 THE CINEMA APPROACH 234 We suggest that each study included in the network is evaluated according to its 235 relevance to the research question and classified into low, moderate or high indirectness. 236 Note that only participant, intervention and outcome characteristics that are likely 237 associated with the relative effect of an intervention against another (that is, effect 238 modifying variables) should be considered. Then, the study-level judgments can be 239 combined with the percentage contribution matrix to produce a bar plot similar to the one 240 presented in Figure 2. Evaluation of indirectness for each relative treatment effect can then 241 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 12 proceed by judging whether the contribution from studies of high or moderate indirectness 242 is important. 243 This approach also addresses the assumption of transitivity in network meta-analysis. 244 Transitivity assumes that we can learn about the relative treatment effect of, say treatment 245 A versus treatment B from an indirect comparison via C. This holds when the distributions of 246 all effect modifiers are comparable in A versus C and B versus C studies. Differences in the 247 distribution of effect modifiers across studies and comparisons will indicate intransitivity. 248 Evaluation of the distribution of effect modifiers is only possible when enough studies are 249 available per comparison. Consequently, the proposed approach will not address 250 intransitivity in sparse networks (when there are few studies compared to the total number 251 of treatments). Assessment of transitivity will be challenging or impossible for interventions 252 that are poorly connected to the network. A further potential obstacle is that details of 253 important effect modifiers might not always be reported in trial reports. For these reasons, 254 we recommend that the network structure and the amount of available data are 255 considered, and that judgments are on the side of caution, as highlighted in the following 256 example. 257 EXAMPLE: COMPARING ANTIDEPRESSANTS 258 Cipriani et al concluded that there is no indirectness in any of the studies included and 259 that the distribution of modifiers was similar across studies and comparisons (12). However, 260 they decided to downgrade evidence about drugs that are poorly connected to the network. 261 For example, vortioxetine was examined in a single study and consequently it was difficult 262 to assess the comparability of effect modifiers in the comparisons with vortioxetine. 263 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 13 Consequently, Cipriani et al. voiced concerns about indirectness for all comparisons with 264 vortioxetine. 265 IMPRECISION 266 BACKGROUND AND DEFINITIONS 267 One of the key advantages of network meta-analysis compared to pairwise meta-268 analysis is the ability to gain precision (36): adding indirect evidence on a particular 269 treatment comparison on top of direct evidence leads to narrower confidence intervals than 270 using the direct evidence alone. However, in network meta-analysis treatment effects are 271 also estimated with uncertainty, typically expressed as 95% confidence intervals that give an 272 indication of where the true effect is likely to lie. To evaluate imprecision it is customary to 273 define relative treatment effects that exclude any clinically important differences in 274 outcomes between interventions (26). At its simplest, this treatment effect might 275 correspond to no effect (0 on an additive scale, 1 on a ratio scale). This would mean that 276 even a small difference is considered important, leading to one treatment being preferred 277 over another. Alternatively, ranges may be defined that divide relative treatment effects 278 into three categories: ‘in favour of A’, ‘no important difference between A and B’, and ‘in 279 favour of B’. The middle range is the ‘range of equivalence’, which includes treatment 280 effects that correspond to clinically unimportant differences between interventions. The 281 range of equivalence can be symmetrical (when a clinically important difference is defined, 282 and its reciprocal constitutes the clinically important difference in the opposite direction) or 283 asymmetrical (when clinically important differences vary by direction of effect). For 284 simplicity, we will assume symmetrical ranges of equivalence. 285 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 14 THE CINEMA APPROACH 286 The approach to imprecision consists of comparing the range of treatment effects 287 included in the 95% confidence interval with the range of equivalence. If the 95% 288 confidence interval extends to differences in treatment effects that would lead to different 289 conclusions, for example covering two or all three of the categories defined above, then the 290 results would be considered imprecise, reducing confidence in the treatment effect 291 estimate. Figure 3 shows a hypothetical forest plot that illustrates the CINeMA rules to 292 assess imprecision of network treatment effect estimates for an odds ratio of 0.8. ‘Major 293 concerns’ are assigned to NMA treatment effects with 95% confidence intervals that cross 294 both limits of the range of equivalence, ‘some concerns’ if only the lower or the upper limit 295 of the range of equivalence is crossed and ‘no concerns’ apply to estimates that do not cross 296 either value. 297 EXAMPLE: ADVERSE EVENTS OF STATINS 298 Consider the network comparing adverse events of different statins, introduced in Box 299 1 and shown in Figure 1C (37). Let us assume a range of equivalence such that an odds ratio 300 greater than 1.05 or below 0.95 ( ) would lead to favouring one the two treatments. Odds 301 ratios between 0.95 and 1.05 would be interpreted as no important differences in the safety 302 profile of the two statins. The 95% confidence interval of pravastatin versus rosuvastatin is 303 quite wide, including odds ratios from 1.09 to 1.82 (Figure 4), but any treatment effect in 304 this range would lead to the conclusion that pravastatin is safer than rosuvastatin. Thus, in 305 this case the imprecision does not reduce the confidence that can be placed in the 306 comparison of pravastatin with rosuvastatin (‘no concerns’). The 95% confidence interval of 307 pravastatin versus simvastatin is slightly wider (0.84 to 1.42) and, more importantly, the 308 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 15 interval covers all three areas, i.e. favouring pravastatin, favouring simvastatin and no 309 important difference. This result is very imprecise, and a rating of ‘major concerns’ applies. 310 The comparison of rosuvastatin versus simvastatin is more certain, but it is again unclear 311 which drug has fewer adverse effects: most estimates within the 95% confidence interval 312 favour simvastatin, but the interval crosses into the range of equivalence. A rating of ‘some 313 concerns’ will be appropriate here. 314 EXAMPLE: EFFICACY OF ANTIDEPRESSANTS 315 In the network of antidepressants, the authors defined clinically important effects as 316 an odds ratio smaller than 0.8 and larger than its reciprocal 1.25 (12). We use this range of 317 equivalence (0.8 to 1.25) in this example. We will concentrate on three comparisons, 318 clomipramine versus fluvoxamine, citalopram versus venlafaxine and amitriptyline versus 319 paroxetine (Table 4). The 95% confidence interval of the odds ratio comparing clomipramine 320 with fluvoxamine (0.75 to 1.32) includes clinically important effects in both directions, 321 implying large uncertainty in which drug should be favored (‘major concerns’) (Table 5). The 322 odds ratio for citalopram versus venlafaxine is 1.12 (95% confidence interval 0.90 to 1.39), 323 favoring venlafaxine, but the interval includes values within the range of equivalence. The 324 verdict therefore is ‘some concerns’. Finally, the odds ratio of amitriptyline versus 325 paroxetine is 0.96 (95% confidence interval 0.82 to 1.13) in favor of amitriptyline. Despite 326 the fact that the estimate includes 1, it is not imprecise because the 95% confidence interval 327 is within the range of equivalence (‘no concerns’). 328 329 330 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 16 HETEROGENEITY 331 BACKGROUND AND DEFINITIONS 332 Variability in the results of studies contributing to a particular comparison influences 333 the confidence we have in the result for that comparison. If this variability reflects genuine 334 differences between studies, rather than random variation, it is usually referred to as 335 heterogeneity. The GRADE system for pairwise meta-analysis uses the term inconsistency to 336 describe such variability (38). In network meta-analysis, there may be variation in the 337 relative treatment effects between studies within a comparison, i.e. heterogeneity, or 338 variation between direct and indirect sources of evidence across comparisons, i.e. 339 incoherence (39–42) which we discuss in the next paragraph. The two notions are closely 340 related; incoherence can be seen as a special form of heterogeneity. 341 There are several ways of measuring heterogeneity in a set of trials. The variance of 342 the distribution of the underlying treatment effects (, is a useful measure of the 343 magnitude of heterogeneity. One can estimate heterogeneity variances from each pairwise 344 meta-analysis and, under the usual assumption of a single variance across comparisons, a 345 common heterogeneity variance for the whole network. The magnitude of  is usefully 346 expressed in a prediction interval, which shows where the true effect of a new study similar 347 to the existing studies is expected to lie (28). 348 THE CINEMA APPROACH 349 Similarly to imprecision, the CINeMA approach to heterogeneity considers its 350 influence on clinical conclusions. Large variability in the included studies does not 351 necessarily affect conclusions, while even small amounts of heterogeneity may be important 352 in some cases. The concordance between assessments based on confidence intervals (which 353 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 17 do not capture heterogeneity) and prediction intervals (which do capture heterogeneity) 354 can be used to assess the importance of heterogeneity. For example, a prediction interval 355 may include values that would lead to different conclusions than suggested by the CI; in 356 such a case, heterogeneity would be considered having important implications. The 357 hypothetical forest plot of Figure 3 serves as an illustration of the CINeMA rules to assess 358 heterogeneity of treatment effects for a clinically important odds ratio of below 0.8 or 359 above 1.25. 360 With only a handful of trials, one cannot adequately estimate the amount of 361 heterogeneity: prediction intervals derived from meta-analyses with very few studies can be 362 unreliable. In this situation it may be more reasonable to interpret an estimate of 363 heterogeneity (and its uncertainty) using empirical distributions. Turner et al. and Rhodes et 364 al. analyzed many meta-analyses of binary and continuous outcomes, categorized them 365 according to the outcome and type of intervention and comparison, and derived empirical 366 distributions of heterogeneity values (16, 17). These empirical distributions can help to 367 interpret the magnitude of heterogeneity, complementary to considerations based on 368 prediction intervals. 369 EXAMPLE: ADVERSE EVENTS OF STATINS 370 In the statins example (Figure 1C), we assumed that the range of equivalence was 371 0.95 to 1.05. The prediction interval of pravastatin versus simvastatin is wide (Figure 4). 372 However, the confidence interval for this comparison already extended into clinically 373 important effects in both directions; thus, the implications of heterogeneity is not important 374 and does not change the conclusion. The confidence interval for pravastatin versus 375 rosuvastatin lies entirely above the equivalence range and is consequently considered 376 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 18 sufficiently precise. However, the corresponding prediction interval crosses both boundaries 377 (0.95 and 1.05), and we therefore would have ‘major concerns’ about the impact of 378 heterogeneity. Similar considerations result in ‘some concerns’ regarding heterogeneity for 379 the comparison rosuvastatin versus simvastatin. 380 EXAMPLE: EFFICACY OF ANTIDEPRESSANTS 381 In the antidepressants network, the estimated amount of heterogeneity is small 382  . The prediction interval for clomipramine versus fluvoxamine does not add 383 further uncertainty to clinical conclusions beyond that already represented by the 384 confidence interval (Table 4), so we have ‘no concerns’ about heterogeneity for that 385 comparison (Table 5). The prediction interval of citalopram versus venlafaxine extend into 386 clinically important effects in both directions (0.74 to 1.70) while the confidence interval 387 does not extend into values in favour of citalopram, thus suggesting potential implications 388 of heterogeneity (‘some concerns’). We have ‘major concerns’ about the impact of 389 heterogeneity for the comparison amitriptyline versus paroxetine, since the confidence 390 interval lies entirely within the range of equivalence, whereas the prediction interval 391 includes clinically important effects in favour of both treatments (0.65, 1.42). 392 INCOHERENCE 393 BACKGROUND AND DEFINITIONS 394 The assumption of transitivity stipulates that we can compare two treatments 395 indirectly via an intermediate treatment node. Incoherence is the statistical manifestation 396 of intransitivity; if transitivity holds, the direct and indirect evidence will be in agreement 397 (45,46). Conversely, if estimates from direct and indirect evidence disagree we conclude 398 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 19 that transitivity does not hold. There are two approaches to quantifying incoherence. The 399 first comprises methods that examine the agreement between direct and indirect evidence 400 for specific comparisons in the network, while the second includes methods that examine 401 incoherence in the entire network. SIDE (Separate Indirect from Direct Evidence) or “node 402 splitting” (39)) is an example of the first set of methods, which are often referred to as local 403 methods. It compares direct and indirect evidence for each comparison and computes an 404 inconsistency factor with a confidence interval. The inconsistency factor is calculated as the 405 difference of the two estimates for an additive measure (e.g. log odds ratio, log risk ratio, 406 standardized mean difference) or as the ratio of the two estimates for measures on the 407 ratio scale. This method can be applied to comparisons that are informed by both direct and 408 indirect evidence. Consider for example the hypothetical example in Figure 3 (Incoherence, 409 Scenario A). The studies directly comparing the two treatments result in a direct odds ratio 410 of 1.75 (1.5 to 2) while the rest studies of the network that provide indirect evidence to the 411 particular comparison gives an indirect odds ratio of 1.37 (1.2 to 1.55). The disagreement 412 between direct and indirect odds ratios is expressed as the ‘inconsistency factor’ (1.27) 413 which can be used to construct a confidence interval (1.05 to 1.55) and a test statistic, here 414 resulting to a p-value of 0.07. A simpler version of SIDE splitting considers a single loop in 415 the network (loop-specific approach (47)). The second set of methods are global methods 416 that model all treatment effects and all possible inconsistency factors simultaneously, 417 resulting in an omnibus test of incoherence in the whole network. The design-by-treatment 418 interaction test is such a global method for incoherence (41,42). An overview of other 419 methods for testing incoherence can be found elsewhere (40,48). 420 421 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 20 THE CINEMA APPROACH 422 Both global and local incoherence tests have low power (49,50) and it is therefore 423 important to consider the inconsistency factors as well as their uncertainty. As a large 424 inconsistency factor may be indicative of a biased direct or indirect estimate, judging its 425 magnitude is always important. As for imprecision and heterogeneity, the CINeMA approach 426 to incoherence considers the impact on clinical conclusions, based on visual inspection of 427 the 95% confidence interval of direct and indirect odds ratios and the range of equivalence. 428 Consider the hypothetical examples in Figure 3 (Incoherence). The inconsistency factor 429 using the SIDE splitting approach is the same for the three examples (1.27 with confidence 430 interval 1.05 to 1.55), but their position relative to the range of equivalence differs and 431 affects the interpretation of incoherence. In the first example, the 95% confidence intervals 432 of both direct and indirect odds ratios lie above the range of equivalence: treatment A is 433 clearly favourable, and there are ‘no concerns’ regarding inconsistency. In the second 434 example, the 95% confidence interval of the indirect odds ratio straddles the range of 435 equivalence while for the direct odds ratio the 95% confidence interval lies entirely above 436 1.05. In this situation, a judgement of ‘some concerns’ is appropriate. In the third example, 437 the odds ratios from direct and indirect comparisons are in opposite directions and the 438 disagreement will therefore lead to an expression of ‘major concerns’. 439 Note that in the three hypothetical examples above, both direct and indirect 440 estimates exist. It could be, however, that there is only direct (e.g. venlafaxine versus 441 vortioxetine in the network of antidepressants) or only indirect (e.g. agomelative versus 442 vortioxetine) evidence. In this situation, we can neither estimate an inconsistency factor nor 443 judge potential implications with respect to the range of equivalence. Considerations of 444 indirectness and intransitivity are nevertheless important. Statistically, incoherence can only 445 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 21 be judged using the global design-by-treatment interaction test. When a comparison is 446 informed only by direct evidence, no disagreement between sources of evidence occurs and 447 thus ‘no concerns’ for incoherence apply. If only indirect evidence is present then there will 448 always be ‘some concerns’. There will be ‘major concerns’ if the p-value of the design-by-449 treatment interaction test is 0.01. As in comparisons informed only by indirect evidence 450 coherence cannot be tested, having ‘no concerns’ for the particular treatment effects would 451 be difficult to defend. 452 EXAMPLE: COMPARING ANTIDEPRESSANTS 453 In the network of antidepressants, the direct odds ratio comparing clomipramine with 454 fluvoxamine is almost double the indirect odds ratio: the ratio of the two odds ratios (i.e., 455 the inconsistency factor) is 1.94 (95% confidence interval 0.65 to 5.73, Table 4). However, 456 both direct and indirect estimates contain values that extend to clinically important values 457 in both directions. Thus, incoherence will not affect the interpretation of the NMA 458 treatment effect: there are ‘no concerns’ (Table 5). In contrast, there are ‘major concerns’ 459 regarding the confidence in the citalopram versus venlafaxine comparison: the direct odds 460 ratio contains values within and above the range of equivalence while the indirect odds 461 ratio includes values within and below the range of equivalence. The resulting estimated 462 ratio of odds ratios is 2.08 (95% confidence interval 1.03 to 4.18) and the respective p-value 463 of the SIDE test is 0.04 (Table 4). For the comparisons of amitriptyline versus paroxetine, the 464 ratio of direct to indirect odds ratios is 1.05 (with 95% confidence interval (0.76, 1.46) and p-465 value 0.75) implying that the two sources of evidence are in agreement (Table 4). Direct and 466 indirect estimates are very close in terms of odds ratios, 95% confidence intervals and the 467 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 22 range of equivalence and we therefore have ‘no concerns’ regarding incoherence for this 468 particular comparison. 469 SUMMARIZING JUDGMENTS ACROSS THE SIX DOMAINS 470 The output of the CINeMA framework is a table with the level of concern for each of 471 the six domains. Some of the domains are interconnected: factors that may reduce the 472 confidence in a treatment effect may affect more than one domain. Indirectness includes 473 considerations on intransitivity, which manifest itself in the data as statistical incoherence. 474 Heterogeneity may be related to most of the other domains. Pronounced heterogeneity will 475 increase imprecision in treatment effects and may be related to variability in within-study 476 biases or the presence of publication bias. Finally, in the presence of heterogeneity the 477 ability to detect important incoherence will decrease (49). 478 Although the final output of CINeMA is a table with the level of concern for each of 479 the six domains, reviewers may choose to summarize judgements across domains. If such an 480 overall assessment is required, one may use the four levels of confidence using the usual 481 GRADE approach: ‘very low’, ‘low’, ‘moderate’ or ‘high’ (24). For this purpose, an initial 482 strategy would be to start at ‘high’ confidence and to drop a rating for each domain with 483 some concerns and to drop two levels for each domain with major concerns. However, the 484 six CINeMA domains should be considered jointly rather than in isolation, avoiding 485 downgrading the overall level of confidence more than once for related concerns. For 486 example, for the ‘citalopram versus venlafaxine’ comparison, we have ‘some concerns’ for 487 imprecision and heterogeneity and ‘major concerns’ for incoherence (Table 3). However, 488 downgrading by two levels will be sufficient in this situation, because imprecision, 489 heterogeneity and incoherence are interconnected. 490 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 23 DISCUSSION 491 We have outlined and illustrated the CINeMA approach for evaluating confidence in 492 treatment effect estimates from NMA, covering the six domains of within-study bias, across-493 study bias, indirectness, imprecision, heterogeneity and incoherence. Our approach avoids 494 selective use of indirect evidence, while considering the characteristics of all studies 495 included in the network. Thus, we are not using assessments of confidence to decide 496 whether to present direct or indirect (or combined) evidence, as has been suggested by 497 others (4,5). We differentiate between the three sources of variability in a network, namely, 498 imprecision, heterogeneity and incoherence and we consider the impact that each source 499 might have on decisions for treatment. The approach can be operationalized and is easy-to-500 implement even for very large networks. 501 Any approach to evaluating confidence in evidence synthesis results will inevitably 502 involve some subjectivity. Our approach is no exception. While the use of bar charts to infer 503 about the impact of within study biases and indirectness provides a consistent assessment 504 across all comparisons in the network, their summary is difficult. Setting up a margin of 505 equivalence might be equivocal. Further limitations of the framework are associated with 506 the fact that published articles are used to make judgements and these reports do not 507 necessarily reflect the way studies were undertaken. For instance, judging indirectness 508 requires study data to be collected on pre-specified effect modifiers; reporting limitations 509 will inevitably impact on the reliability of the judgements. 510 A consequence of the inherent subjectivity of the system is that interrater 511 agreement may be modest. Studies of the reproducibility of assessments made by 512 researchers using CINeMA will be required in this context. We believe however that 513 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 24 transparency is key. Although judgements may differ across reviewers, they are made using 514 explicit criteria. These should be specified in the review protocol so that data-driven 515 decisions are avoided. The web application at cinema.ispm.ch will greatly facilitate the 516 implementation of all steps involved in the application of CINeMA (6). 517 This paper proposes a refinement of a previously suggested framework (51). An 518 alternative approach has also been refined (52) since its initial introduction (53). The two 519 methods have similarities but also notable differences. For example, Puhan et al (53) 520 suggest a process of deciding whether indirect estimates are of sufficient certainty to 521 combine them with the direct estimates. In contrast CINeMA evaluates relative treatment 522 effects without considering separately the direct and indirect sources. Evaluation of the 523 impact of within-study bias and indirectness differs materially between the two approaches. 524 The need to choose the most influential one-step loop in the GRADE approach as described 525 by Puhan et al. (53) and Brignardello-Petersen (18) discards a large amount of information 526 that contributes to the results and makes the approach difficult to apply to large networks. 527 The percentage contribution matrix appears to be the only viable option to acknowledge 528 the impact of each and every study included in a network. Moreover, our framework 529 naturally includes the results from sensitivity analyses in the interpretation of the bar 530 charts. Finally, in contrast to GRADE approach, we do not rely on metrics for judging 531 heterogeneity and incoherence: we consider instead the impact that these can have when a 532 stakeholder needs to make informed decisions. An alternative approach to assessing 533 confidence findings from network meta-analysis is to explore how robust treatment 534 recommendations are to potential degrees of bias in the evidence (54). The method is easy 535 to apply but focuses on the impact of bias and does not explicitly address heterogeneity, 536 indirectness and inconsistency. 537 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 25 Evidence synthesis is increasingly used by national and international agencies (55,56) 538 to inform decisions about the reimbursement of medical interventions, by clinical guideline 539 panels to recommend one drug over another and by clinicians to prescribe a treatment or 540 recommend a diagnostic procedure for individual patients. However, it is the exception 541 rather than the rule for published network meta-analyses to formally evaluate confidence in 542 relative treatment effects (57). With the use of open-source free software (see Box 2), our 543 approach can be routinely applied to any network meta-analysis (6) and offers a step 544 forward in transparency and reproducibility. The suggested framework operationalizes, 545 simplifies and accelerates the process of evaluation of results from large and complex 546 networks without compromising in statistical and methodological rigor. The CINeMA 547 framework is a transparent, rigorous and comprehensive system for evaluating the 548 confidence of treatment effect estimates from network meta-analysis. 549 550 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 26 Box 1. Description of three network meta-analyses used to illustrate the CINeMA 551 approach to assess confidence in network meta-analysis. 552 Diagnostic strategies for patients with low risk of acute coronary syndrome 553 Siontis et al performed a network meta-analysis to of randomized trials to evaluate the 554 differences between the non-invasive diagnostic modalities used to detect coronary artery 555 disease in patients presenting with symptoms suggestive of acute coronary syndrome (11). 556 Differences between the diagnostic modalities were evaluated with respect to the number 557 of downstream referrals for invasive coronary angiography and other clinical outcomes. For 558 outcome referrals, 18 studies were included. The network is presented in Figure 1A and the 559 data in Table S1. The results from the network meta-analysis are presented in Table 1. 560 Antidepressants for moderate and major depression 561 Cipriani et al compared 18 commonly prescribed antidepressants, which were studied in 562 179 head-to-head randomized trials involving patients diagnosed with major/moderate 563 depression (12). The primary efficacy outcome was response measured as 50% reduction in 564 the symptoms scale between baseline and 8 weeks of follow-up. According to the inclusion 565 criteria specified in the protocol only studies at low or moderate risk of bias were included 566 (58). The methodological and statistical details presented in the published article and its 567 appendix. Here, we will focus on how judgements about credibility of the results were 568 derived. The network is presented in Figure 1B and the data is available in Mendeley Data 569 (DOI:10.17632/83rthbp8ys.2). 570 Comparative tolerability and harms of statins 571 The aim of the systematic review by Naci et al. (37) was to determine the comparative 572 tolerability and harms of eight statins. The outcome considered here is the number of 573 patients who discontinued therapy due to adverse effects, measured as an odds ratio. This 574 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 27 outcome was evaluated in 101 studies. The network is presented in Figure 1C and the 575 outcome data are given in Table S4. The results of the network meta-analysis are presented 576 in Table S5 and the results from SIDE splitting in Table S6. 577 578 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 28 Box 2. Description of the CINeMA web-application. 579 THE CINeMA WEB APPLICATION 580 CINeMA framework has been implemented in a freely available, user-friendly web-581 application aiming to facilitate the evaluation of confidence on the results from network 582 meta-analysis (http://cinema.ispm.ch/ (6)). The web application is programmed in 583 javascript, uses docker and is linked with R; in particular, packages meta and netmeta are 584 used (59). Knowledge of the aforementioned languages and technologies is however not 585 required from the users. 586 Loading the data 587 In ‘My Projects’ tab, CINeMA users are able to upload a .csv file with the by-treatment 588 outcome study data and study-level risk of bias (RoB) and indirectness judgments. CINeMA 589 web-application can handle all the formats used in network meta-analysis (long or wide 590 format, binary or continuous, arm level or study level data) and provides flexibility in 591 labelling variables as desired by the user. A demo dataset is available in ‘My Projects’ tab. 592 Evaluating the confidence in the results from network meta-analysis 593 A preview of the evidence (network plot and outcome data) and options concerning the 594 analysis (fixed or random effects, effect measure etc.) are given in the ‘Configuration’ tab. 595 The next six tabs guide users to make informed conclusions on the quality of evidence based 596 on within-study bias, across-studies bias, indirectness, imprecision, heterogeneity and 597 incoherence. Features implemented include the percentage contribution matrix, relative 598 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 29 treatment effects for each comparison, estimation of the heterogeneity variance, prediction 599 intervals and tests for the evaluation of the assumption of coherence. 600 Summarising judgments 601 The last tab ‘Report’ includes a summary of the evaluations made in the six domains and 602 gives users the possibility to either not downgrade, or downgrade by one or two levels each 603 relative treatment effect. Users can download a report with the summary of their 604 evaluations along with their final judgements. CINeMA is accompanied by a documentation 605 describing each step in detail (tab ‘Documentation’). 606 607 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 30 608 Table 1. Results from pairwise (upper triangle) and network meta-analysis (lower triangle) from 609 the network of non-invasive diagnostic strategies for the detection of coronary artery disease in 610 Figure 1A. Odds ratios and their 95% confidence intervals are presented for referrals for invasive 611 coronary angiography. Odds ratios in the lower triangle less than one favor the strategy in the 612 column; odds ratios in the upper triangle less than one favor the strategy in the row. Cells with a 613 dot indicate that no direct studies examine the particular comparison. 614 CCTA . 2.25 [1.04 - 4.90] 1.04 [0.70 - 1.55] 1.23 [1.00 - 1.50] . 3.07 [1.46 - 6.45] CMR . . 0.38 [0.18 - 0.78] . 2.24 [1.22 - 4.11] 0.73 [0.28 - 1.88] Exercise ECG . 0.42 [0.14 - 1.30] 1.93 [1.39 - 2.67] 1.27 [1.01 - 1.60] 0.42 [0.20 - 0.87] 0.57 [0.30 - 1.07] SPECT-MPI 0.87 [0.71 - 1.06] . 1.17 [0.97 - 1.40] 0.38 [0.18 - 0.78] 0.52 [0.28 - 0.96] 0.92 [0.76 - 1.10] Standard Care 2.95 [0.97 - 8.98] 4.31 [2.23 - 8.32] 1.40 [0.53 - 3.74] 1.93 [1.39 - 2.66] 3.38 [1.71 - 6.68] 3.69 [1.90 - 7.17] Stress Echo ECG: electrocardiogram; echo: echocardiography; SPECT-MPI: single photon emission computed 615 tomography-myocardial perfusion imaging; CCTA: coronary computed tomographic angiography; 616 CMR: cardiovascular magnetic resonance. 617 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 31 Table 2. The percentage contribution matrix for the network presented in Figure 1A. The columns refer to the studies (grouped by comparison) and the rows refer to the relative treatment effects (grouped into mixed and indirect evidence) from network meta-analysis. The entries show how much each study contributes (as percentage) to the estimation of relative treatment effects. Direct comparisons (number of studies) CCTA vs Exercise ECG (1) CCTA vs SPECT-MPI (2) CCTA vs Standard care (7) CMR vs Standard care (2) Exercise ECG vs Standard care (1) Exercise ECG vs Stress Echo (4) SPECT-MPI vs Standard care (2) Standard care vs Stress Echo (1) NMA Estimates/study IDs 3 2 9 1 10 13 14 4 7 8 11 6 12 12 15 16 17 18 5 12 Mixed estimates CCTA:Exercise ECG 52 1 1 3 0 3 1 3 4 4 0 0 14 0 3 0 2 1 1 6 CCTA:SPECT-MPI 1 18 16 5 1 5 1 6 7 7 0 0 0 0 0 0 0 22 10 0 CCTA:Standard care 1 4 4 13 2 13 3 15 18 17 0 0 1 0 0 0 0 6 3 0 CMR:Standard care 0 0 0 0 0 0 0 0 0 0 60 40 0 0 0 0 0 0 0 0 Exercise ECG:Standard care 23 1 1 3 0 3 1 4 5 4 0 0 30 1 6 1 3 2 1 11 Exercise ECG:Stress Echo 1 0 0 0 0 0 0 0 0 0 0 0 1 5 52 8 29 0 0 2 SPECT-MPI:Standard care 0 5 4 1 0 1 0 2 2 2 0 0 0 0 0 0 0 57 26 0 Standard care:Stress Echo 14 1 1 2 0 2 1 2 3 3 0 0 14 2 16 2 9 1 1 27 Indirect estimates -- -- -- -- -- -- -- -- -- -- -- -- 0 0 -- -- -- -- -- 0 CCTA:CMR 1 3 2 6 1 7 2 8 9 8 28 19 1 0 0 0 0 4 2 0 CCTA:Stress Echo 24 1 1 3 0 3 1 3 4 4 0 0 8 2 18 3 10 1 1 13 CMR:Exercise ECG 16 1 1 2 0 2 1 3 3 3 22 15 15 0 4 1 2 1 1 7 CMR:SPECT-MPI 0 3 3 1 0 1 0 1 1 1 28 19 0 0 0 0 0 28 13 0 CMR:Stress Echo 11 1 1 1 0 2 0 2 2 2 20 14 9 1 11 2 6 1 0 13 Exercise ECG:SPECT-MPI 21 7 6 1 0 1 0 1 2 2 0 0 15 0 4 1 2 20 9 7 SPECT-MPI:Stress Echo 14 5 4 1 0 1 0 1 1 1 0 0 9 1 13 2 7 19 9 13 ECG: electrocardiogram; echo: echocardiography; SPECT-MPI: single photon emission computed tomography-myocardial perfusion imaging; CCTA: coronary computed tomographic angiography; CMR: cardiovascular magnetic resonance. .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 32 Table 3. Summary odds ratios from network meta-analysis comparing six antidepressants and sensitivity analyses excluding studies at moderate risk of bias. Comparison Response odds ratio [95% confidence interval] All studies (179 studies) Studies at low risk of bias (83 studies) Amitriptyline versus Milnacipran 1.11 [0.85; 1.43] 1.10 [0.77; 1.59] Mirtazapine versus Paroxetine 1.07 [0.88; 1.30] 1.08 [0.83; 1.39] Amitriptyline versus Clomipramine 1.24 [0.97; 1.59] 0.96 [0.59; 1.57] .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 33 Table 4. Results from direct, indirect and mixed evidence along with confidence and prediction intervals and incoherence ratio of odds ratios for the network of antidepressants. Odds ratios lower than 1 favour the first treatment. Comparison Direct OR (95% CI) Indirect OR (95% CI) Ratio of ORs (95% CI) NMA OR (95% CI) 95% PrI of NMA OR Clomipramine versus Fluvoxamine 1.85 (0.65 to 5.27) 0.96 (0.71 to 1.29) 1.94 (0.65 to 5.73) 0.99 (0.75 to 1.32) (0.63 to 1.57) Citalopram versus Venlafaxine 1.72 (0.89 to 3.32) 0.83 (0.66 to 1.04) 2.08 (1.03 to 4.18) 1.12 (0.90 to 1.39) (0.74 to 1.70) Amitriptyline versus Paroxetine 1.07 (0.85 to 1.36) 1.02 (0.82 to 1.27) 1.05 (0.76 to 1.46) 0.96 (0.82 to 1.13) (0.65 to 1.42) NMA: network meta-analysis, OR: odds ratio, PrI: prediction interval, CI: confidence interval. .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 34 Table 5. Level of concern for three network meta-analysis odds ratios from the network of antidepressants for the domains imprecision, heterogeneity and incoherence. See Table 4 for odds ratios. Comparison Imprecision Heterogeneity Incoherence Clomipramine versus Fluvoxamine Major concerns No concerns No concerns Citalopram versus Venlafaxine Some concerns Some concerns Major concerns Amitriptyline versus Paroxetine No concerns Major concerns No concerns .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 35 Figure 1. Network plots of the three network meta-analyses used as examples. The width of the edges are proportional to the number of patients randomised in each comparison. A: Network of randomised controlled trials comparing non-invasive diagnostic strategies for the detection of coronary artery disease in patients with low risk acute coronary syndrome. The colours of edges and nodes refer to the risk of bias; low (green), moderate (yellow) and red (high). In square brackets are the study IDs as presented in Table S1. B: Network of randomised controlled trials comparing active antidepressants in patients with moderate/major depression. The colours of edges refer to the risk of bias; low (green), moderate (yellow) and red (high). The size of nodes is proportional to the number of studies examining each treatment. C: Network of randomised controlled trials comparing statins with respect to adverse effects. ECG: electrocardiogram; echo: echocardiography; SPECT-MPI: single photon emission computed tomography-myocardial perfusion imaging; CCTA: coronary computed tomographic angiography; CMR: cardiovascular magnetic resonance. .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 36 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 37 Figure 2. Risk of bias bar chart for the comparison of non-invasive diagnostic strategies for the detection of coronary artery disease. Each bar represents a relative treatment effect estimated using the data in the network in Error! Reference source not found.A. White vertical lines indicate the percentage contribution of separate studies. Each bar shows the percentage contribution from studies judged to be at low (green), moderate (yellow) and high (red) risk of bias. ECG: electrocardiogram; echo: echocardiography; SPECT-MPI: single photon emission computed tomography-myocardial perfusion imaging; CCTA: coronary computed tomographic angiography; CMR: cardiovascular magnetic resonance. .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 38 Figure 3. CINeMA rules to assess imprecision, heterogeneity and incoherence of network treatment effects. The range of equivalence is from 0.8 to 1.25. Black lines indicate confidence intervals and red lines indicate prediction intervals. For the three scenarios presented for incoherence, inconsistency factor is 1.27 (1.05 to 1.55). OR: odds ratio. .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 39 Figure 4. Network meta-analysis odds ratios from the network of statins their 95% confidence intervals (black lines) and their 95% prediction intervals (red lines). The range of equivalence is from 0.95 to 1.05. PrI: 95% prediction interval, CI: 95% confidence interval, vs: versus. .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 40 Acknowledgements The development of the software and part of the presented work was supported by the Campbell Collaboration. ME was supported by special project funding (Grant No. 174281) from the Swiss National Science Foundation. GS, AN, TP were supported by project funding (Grant No. 179158) from the Swiss National Science Foundation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 41 REFERENCES 1. Zarin W, Veroniki AA, Nincic V, Vafaei A, Reynen E, Motiwala SS, et al. Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review. BMC Med. 2017 Jan 5;15(1):3. 2. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008 Apr 26;336(7650):924–6. 3. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011 Apr;64(4):383–94. 4. Puhan MA, Schünemann HJ, Murad MH, Li T, Brignardello-Petersen R, Singh JA, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014;349:g5630. 5. Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JPT. Evaluating the quality of evidence from a network meta-analysis. PloS One. 2014;9(7):e99682. 6. CINeMA: Confidence in Network Meta-Analysis. [Internet]. Institute of Social and Preventive Medicine, University of Bern.; 2017. Available from: cinema.ispm.ch 7. Jansen JP, Trikalinos T, Cappelleri JC, Daw J, Andes S, Eldessouki R, et al. Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health J Int Soc Pharmacoeconomics Outcomes Res. 2014 Mar;17(2):157–73. 8. Hoaglin DC, Hawkins N, Jansen JP, Scott DA, Itzler R, Cappelleri JC, et al. Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. ValueHealth. 2011 Jun;14(1524–4733 (Electronic)):429–37. 9. Ades AE, Caldwell DM, Reken S, Welton NJ, Sutton AJ, Dias S. Evidence synthesis for decision making 7: a reviewer’s checklist. MedDecisMaking. 2013 Jul;33(1552–681X (Electronic)):679–91. 10. Dias S, Sutton AJ, Ades AE, Welton NJ. Evidence synthesis for decision making 2: a generalized linear modeling framework for pairwise and network meta-analysis of randomized controlled trials. Med Decis Mak Int J Soc Med Decis Mak. 2013 Jul;33(5):607–17. 11. Siontis GC, Mavridis D, Greenwood JP, Coles B, Nikolakopoulou A, Jüni P, et al. Outcomes of non-invasive diagnostic modalities for the detection of coronary artery disease: network meta-analysis of diagnostic randomised controlled trials. BMJ. 2018 Feb 21;360:k504. .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 42 12. Cipriani A, Furukawa TA, Salanti G, Chaimani A, Atkinson LZ, Ogawa Y, et al. Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta-analysis. Lancet Lond Engl. 2018 Feb 20; 13. Naci H, Brugts J, Ades T. Comparative tolerability and harms of individual statins: a study-level network meta-analysis of 246 955 participants from 135 randomized, controlled trials. Circ Cardiovasc Qual Outcomes. 2013 Jul;6(4):390–9. 14. Rücker G, Schwarzer G, Krahn U, Konig J. netmeta: Network Meta-Analysis using Frequentist Methods. R package version 0.8-0. [Internet]. Available from: https://CRAN.R-project.org/package=netmeta 15. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011 Oct 18;343:d5928. 16. Higgins JP, Sterne J, Savovic J, Page MJ, Hróbjartsson A, Boutron I, et al. A revised tool for assessing risk of bias in randomized trials. In: Cochrane Methods Cochrane Database of Systematic Reviews. Chandler J, McKenzie J, Boutron I, Welch V; 2016. (Issue 10 (Suppl 1)). 17. Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, et al. GRADE guidelines: 4. Rating the quality of evidence--study limitations (risk of bias). J Clin Epidemiol. 2011 Apr;64(4):407–15. 18. Brignardello-Petersen R, Bonner A, Alexander PE, Siemieniuk RA, Furukawa TA, Rochwerg B, et al. Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis. J Clin Epidemiol. 2018 Jan;93:36–44. 19. Papakonstantinou T, Nikolakopoulou A, Rücker G, Chaimani A, Schwarzer G, Egger M, et al. Estimating the contribution of studies in network meta-analysis: paths, flows and streams. F1000Research. 2018 May 18;7:610. 20. Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014 Oct 1;(10):MR000035. 21. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PloS One. 2008 Aug 28;3(8):e3081. 22. Dwan K, Gamble C, Williamson PR, Kirkham JJ, Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PloS One. 2013;8(7):e66844. 23. Scherer RW, Langenberg P, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000005. .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 43 24. Wager E, Williams P, Project Overcome failure to Publish nEgative fiNdings Consortium. “Hardly worth the effort”? Medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: quantitative and qualitative study. BMJ. 2013 Sep 6;347:f5248. 25. Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997 Sep 13;315(7109):640–5. 26. Dickersin K, Chalmers I. Recognizing, investigating and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the WHO. J R Soc Med. 2011 Dec;104(12):532–8. 27. Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. Rating the quality of evidence--publication bias. J Clin Epidemiol. 2011 Dec;64(12):1277–82. 28. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003 May 31;326(7400):1167–70. 29. Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine--selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ. 2003 May 31;326(7400):1171–3. 30. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008 Jan 17;358(3):252–60. 31. Chaimani A, Salanti G. Using network meta-analysis to evaluate the existence of small-study effects in a network of interventions. Res Synth Methods. 2012;3(2):161–176. 32. Chaimani A, Salanti G. Visualizing assumptions and results in network meta-analysis: The network graphs package. 2015;15(4):905–50. 33. Mavridis D, Efthimiou O, Leucht S, Salanti G. Publication bias and small-study effects magnified effectiveness of antipsychotics but their relative ranking remained invariant. J Clin Epidemiol. 2015 Jun 5; 34. Mavridis D, Sutton A, Cipriani A, Salanti G. A fully Bayesian application of the Copas selection model for publication bias extended to network meta-analysis. Stat Med. 2013 Jan 15;32(1):51–66. 35. Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 8. Rating the quality of evidence--indirectness. J Clin Epidemiol. 2011 Dec;64(12):1303–10. 36. Caldwell DM, Ades AE, Higgins JPT. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ. 2005 Oct 15;331(7521):897–900. .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 44 37. Naci H, Brugts J, Ades T. Comparative tolerability and harms of individual statins: a study-level network meta-analysis of 246 955 participants from 135 randomized, controlled trials. Circ Cardiovasc Qual Outcomes. 2013 Jul;6(4):390–9. 38. Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 7. Rating the quality of evidence--inconsistency. J Clin Epidemiol. 2011 Dec;64(12):1294–302. 39. Dias S, Welton NJ, Caldwell DM, Ades AE. Checking consistency in mixed treatment comparison meta-analysis. Stat Med. 2010 Mar 30;29(7–8):932–44. 40. Dias S, Welton NJ, Sutton AJ, Caldwell DM, Lu G, Ades AE. Evidence synthesis for decision making 4: inconsistency in networks of evidence based on randomized controlled trials. Med Decis Mak Int J Soc Med Decis Mak. 2013;33(5):641–56. 41. White IR, Barrett JK, Jackson D, Higgins JPT. Consistency and inconsistency in network meta-analysis: model estimation using multivariate meta-regression. Res Synth Methods. 2012 Jun;3(2):111–25. 42. Higgins JPT, Jackson D, Barrett JK, Lu G, Ades AE, White IR. Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies. Res Synth Methods. 2012 Jun;3(2):98–110. 43. Turner RM, Davey J, Clarke MJ, Thompson SG, Higgins JP. Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews. Int J Epidemiol. 2012 Jun;41(3):818–27. 44. Rhodes KM, Turner RM, Higgins JPT. Predictive distributions were developed for the extent of heterogeneity in meta-analyses of continuous outcome data. J Clin Epidemiol. 2015 Jan;68(1):52–60. 45. Lu G, Ades AE. Assessing Evidence Inconsistency in Mixed Treatment Comparisons. J Am Stat Assoc. 2006 Jun 1;101(474):447–59. 46. Lu G, Ades A. Modeling between-trial variance structure in mixed treatment comparisons. Biostat Oxf Engl. 2009 Oct;10(4):792–805. 47. Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol. 1997 Jun;50(6):683–91. 48. Dias S, Welton NJ, Sutton AJ, Caldwell DM, Lu G, Ades AE. Inconsistency in Networks of Evidence Based on Randomised Controlled Trials [Internet]. London: National Institute for Health and Care Excellence (NICE); 2014. (NICE Decision Support Unit Technical Support Documents). Available from: http://www.ncbi.nlm.nih.gov/books/NBK310372/ .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 45 49. Veroniki AA, Mavridis D, Higgins JP, Salanti G. Characteristics of a loop of evidence that affect detection and estimation of inconsistency: a simulation study. BMC Med Res Methodol. 2014 Sep 19;14:106. 50. Song F, Clark A, Bachmann MO, Maas J. Simulation evaluation of statistical properties of methods for indirect and mixed treatment comparisons. BMC Med Res Methodol. 2012 Sep 12;12:138. 51. Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JPT. Evaluating the quality of evidence from a network meta-analysis. PloS One. 2014;9(7):e99682. 52. Brignardello-Petersen R, Bonner A, Alexander PE, Siemieniuk RA, Furukawa TA, Rochwerg B, et al. Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis. J Clin Epidemiol. 2018 Jan;93:36–44. 53. Puhan MA, Schünemann HJ, Murad MH, Li T, Brignardello-Petersen R, Singh JA, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014 Sep 24;349:g5630. 54. Caldwell DM, Ades AE, Dias S, Watkins S, Li T, Taske N, et al. A threshold analysis assessed the credibility of conclusions from network meta-analysis. J Clin Epidemiol. 2016 Dec;80:68–76. 55. Kanters S, Ford N, Druyts E, Thorlund K, Mills EJ, Bansback N. Use of network meta-analysis in clinical guidelines. Bull World Health Organ. 2016 Oct 1;94(10):782–4. 56. Petropoulou M, Nikolakopoulou A, Veroniki A-A, Rios P, Vafaei A, Zarin W, et al. Bibliographic study showed improving statistical methodology of network meta-analyses published between 1999 and 2015. J Clin Epidemiol. 2017 Feb;82:20–8. 57. Zarin W, Veroniki AA, Nincic V, Vafaei A, Reynen E, Motiwala SS, et al. Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review. BMC Med. 2017 05;15(1):3. 58. Furukawa TA, Salanti G, Atkinson LZ, Leucht S, Ruhe HG, Turner EH, et al. Comparative efficacy and acceptability of first-generation and second-generation antidepressants in the acute treatment of major depression: protocol for a network meta-analysis. BMJ Open. 2016;6(7):e010919. 59. Rücker G, Schwarzer G, Krahn U, König J. netmeta: Network Meta-Analysis using Frequentist Methods [Internet]. 2016 [cited 2016 Sep 5]. Available from: https://cran.r-project.org/web/packages/netmeta/index.html .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 46 SUPPLEMENTARY MATERIAL Table S1 Data from Network of randomised controlled trials comparing non-invasive diagnostic strategies for the detection of coronary artery disease in patients with low risk acute coronary syndrome. The data was originally published by Siontis et al. id trial group n r rob t 1 BEACONR1 Anatomical testing 250 41 1 CCTA 1 BEACONR1 Standard care 250 31 1 Standard care 2 Levsky JM., et al.R2 Anatomical testing 200 30 1 CCTA 2 Levsky JM., et al.R2 Functional testing 200 32 1 SPECT-MPI 3 CT-COMPARER3 Anatomical testing 322 26 3 CCTA 3 CT-COMPARER3 Functional testing 240 9 3 Exercise ECG 4 CATCHR4,R5 Anatomical testing 299 49 3 CCTA 4 CATCHR4,R5 Standard care 301 36 3 Standard care 5 Lim SH., et al.R6 Functional testing 1126 73 2 SPECT-MPI 5 Lim SH., et al.R6 Standard care 564 56 2 Standard care 6 Miller CD., et al.R7 CMR 52 5 2 CMR 6 Miller CD., et al.R7 Standard care 53 11 2 Standard care 7 ROMICAT-IIR8 Anatomical testing 501 59 3 CCTA 7 ROMICAT-IIR8 Standard care 499 40 3 Standard care 8 ACRIN-PAR9,R10 Anatomical testing 929 69 1 CCTA 8 ACRIN-PAR9,R10 Standard care 463 32 1 Standard care 9 CT-STATR11 Anatomical testing 375 26 1 CCTA 9 CT-STATR11 Functional testing 374 22 1 SPECT-MPI 10 Miller AH., et al.R12 Anatomical testing 30 4 2 CCTA 10 Miller AH., et al.R12 Standard care 30 4 2 Standard care 11 Miller CD., et al.R13,R14 CMR 52 8 3 CMR 11 Miller CD., et al.R13,R14 Standard care 57 19 3 Standard care .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 47 12 Nucifora G., et al.R15 Functional testing 77 5 1 Stress Echo 12 Nucifora G., et al.R15 Functional testing 75 9 1 Exercise ECG 12 Nucifora G., et al.R15 Standard care 55 8 1 Standard care 13 Chang SA., et al.R16 Anatomical testing 133 47 1 CCTA 13 Chang SA., et al.R16 Standard care 133 57 1 Standard care 14 Goldstein JA., et al.R17 Anatomical testing 99 12 1 CCTA 14 Goldstein JA., et al.R17 Standard care 98 7 1 Standard care 15 Jeetley P., et al.R18 Functional testing 215 41 1 Stress Echo 15 Jeetley P., et al.R18 Functional testing 218 72 1 Exercise ECG 16 Nucifora G., et al.R19 Functional testing 110 6 2 Stress Echo 16 Nucifora G., et al.R19 Functional testing 89 6 2 Exercise ECG 17 Jeetley P., et al.R20 Functional testing 148 21 2 Stress Echo 17 Jeetley P., et al.R20 Functional testing 154 36 2 Exercise ECG 18 Udelson JE., et al.R21 Functional testing 1215 156 2 SPECT-MPI 18 Udelson JE., et al.R21 Standard care 1260 162 2 Standard care .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 48 Table S2 Number of one-step loops providing indirect evidence for NMA relative treatment effects between treatment comparisons for the network of antidepressants. Number of “one-step loops” providing indirect evidence Nr of treatment comparisons Cumulative frequency % Cumulative frequency 0 3 3 2% 1 16 19 14% 2 18 37 28% 3 32 69 51% 4 30 99 74% 5 19 118 88% 6 13 131 98% 7 13 144 107% 8 3 147 110% 10 1 148 110% 11 1 149 111% 12 3 152 113% 13 1 153 114% .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 49 Table S3 Average contribution to NMA relative treatment effects from direct evidence and indirect evidence via intermediate comparators (steps). The “one-step loop” provides one-step indirect comparison via a single common treatment. Source of evidence % Contribution Direct evidence 11.1% Indirect evidence 1 step 56.1% 2 steps 84.3% 3 steps 95.0% 4 steps 98.0% 5 steps 99.1% 6 steps 99.5% 7 steps 99.7% 8 steps 99.9% 9 steps 100.0% 10 steps 100.0% 11 steps 100.0% 12 steps 100.0% 13 steps 100.0% .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 50 Table S4. Data from the network of randomised controlled trials comparing adverse effects of statins. The data was originally published by Naci et al. id: id of the study, t: treatment name, r: number of adverse effects, n: sample size. year study id t r n 1993 PMSG 1 pravastatin 25 530 1993 PMSG 1 placebo 33 532 1993 SPSG 2 simvastatin 5 275 1993 SPSG 2 pravastatin 5 275 1993 LPSG 3 lovastatin 10 339 1993 LPSG 3 pravastatin 8 333 1993 MARS 4 lovastatin 3 123 1993 MARS 4 placebo 6 124 1994 4S 5 placebo 129 2223 1994 4S 5 simvastatin 126 2221 1994 PMSG-Diabetes 6 pravastatin 2 167 1994 PMSG-Diabetes 6 placebo 9 158 1994 EXCEL 7 placebo 100 1663 1994 EXCEL 7 lovastatin 329 6582 1994 OCS 8 simvastatin 18 414 1994 OCS 8 placebo 6 207 1995 Jacobson 9 pravastatin 9 182 1995 Jacobson 9 placebo 1 63 1995 REGRESS 10 pravastatin 16 450 1995 REGRESS 10 placebo 10 434 1995 KAPS 11 placebo 12 223 1995 KAPS 11 pravastatin 8 224 1995 WOSCOPS 12 placebo 106 3293 1995 WOSCOPS 12 pravastatin 116 3302 1995 Guillen 13 placebo 1 74 1995 Guillen 13 pravastatin 0 76 1996 SHIGA Pravastatin study 14 pravastatin 2 102 1996 SHIGA Pravastatin study 14 placebo 0 105 1996 CARE 15 placebo 74 2078 1996 CARE 15 pravastatin 45 2081 1996 QLMG 16 lovastatin 3 211 1996 QLMG 16 pravastatin 4 215 1996 CHESS 17 simvastatin 27 453 1996 CHESS 17 atorvastatin 65 464 1997 Bertolini 18 atorvastatin 7 227 1997 Bertolini 18 pravastatin 2 78 1997 ASG-I 19 atorvastatin 16 529 1997 ASG-I 19 lovastatin 5 120 1998 AFCAPS-TexCAPS 20 placebo 455 3301 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 51 1998 AFCAPS-TexCAPS 20 lovastatin 449 3304 1998 Brown 21 atorvastatin 3 78 1998 Brown 21 fluvastatin 4 76 1998 Brown 21 lovastatin 2 78 1998 Brown 21 simvastatin 2 76 1999 TARGET TANGIBLE 22 atorvastatin 89 1897 1999 TARGET TANGIBLE 22 simvastatin 45 959 1999 Riegger 23 fluvastatin 11 187 1999 Riegger 23 placebo 8 178 1999 IQLMG 24 simvastatin 7 194 1999 IQLMG 24 pravastatin 7 193 1999 FLARE 25 fluvastatin 4 409 1999 FLARE 25 placebo 11 425 2000 Barter 26 atorvastatin 48 691 2000 Barter 26 simvastatin 24 337 2000 Farnier 27 atorvastatin 1 109 2000 Farnier 27 simvastatin 1 163 2000 Stein 28 placebo 0 130 2000 Stein 28 simvastatin 1 260 2000 Recto 29 simvastatin 1 251 2000 Recto 29 atorvastatin 5 251 2000 Gentile 30 atorvastatin 1.5 85 2000 Gentile 30 simvastatin 0.5 79 2000 Gentile 30 pravastatin 1.5 82 2000 Gentile 30 lovastatin 1.5 81 2000 Gentile 30 placebo 0.5 87 2001 ASSET 31 atorvastatin 7 730 2001 ASSET 31 simvastatin 7 694 2001 MIRACL 32 placebo 33 1548 2001 MIRACL 32 atorvastatin 40 1538 2001 Paoletti 33 rosuvastatin 8 230 2001 Paoletti 33 pravastatin 3 136 2001 Paoletti 33 simvastatin 1 129 2001 Andrews 34 atorvastatin 129 1902 2001 Andrews 34 fluvastatin 64 477 2001 Andrews 34 lovastatin 42 476 2001 Andrews 34 pravastatin 20 462 2001 Andrews 34 simvastatin 39 468 2002 GREACE 35 atorvastatin 6 800 2002 GREACE 35 placebo 3 800 2002 Davidson 36 placebo 3 70 2002 Davidson 36 simvastatin 14 263 2002 FLORIDA 37 fluvastatin 30 265 2002 FLORIDA 37 placebo 37 275 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 52 2002 LIPS 38 fluvastatin 174 844 2002 LIPS 38 placebo 196 833 2002 PROSPER 39 placebo 116 1913 2002 PROSPER 39 pravastatin 107 2891 2002 Olsson 40 rosuvastatin 16 272 2002 Olsson 40 atorvastatin 12 140 2002 Davidson 41 placebo 7 132 2002 Davidson 41 rosuvastatin 10 259 2002 Davidson 41 atorvastatin 4 128 2002 CHALLENGE 42 atorvastatin 17 846 2002 CHALLENGE 42 simvastatin 10 848 2003 Ballantyne 43 atorvastatin 13 248 2003 Ballantyne 43 placebo 3 60 2003 ADVOCATE 44 atorvastatin 6 82 2003 ADVOCATE 44 simvastatin 2 76 2003 Bruckert 45 fluvastatin 13 607 2003 Bruckert 45 placebo 8 622 2003 Kerzner 46 lovastatin 10 220 2003 Kerzner 46 placebo 5 64 2003 Melani 47 pravastatin 3 205 2003 Melani 47 placebo 5 65 2003 TREAT TO TARGET 48 atorvastatin 20 552 2003 TREAT TO TARGET 48 simvastatin 14 535 2003 HeFH 49 rosuvastatin 16 436 2003 HeFH 49 atorvastatin 6 187 2003 Mohler 50 atorvastatin 16 240 2003 Mohler 50 placebo 10 114 2003 Davidson 51 lovastatin 21 501 2003 Davidson 51 fluvastatin 22 337 2003 STELLAR 52 rosuvastatin 9 480 2003 STELLAR 52 atorvastatin 25 641 2003 STELLAR 52 simvastatin 19 655 2003 STELLAR 52 pravastatin 11 492 2004 CARDS 53 placebo 145 1410 2004 CARDS 53 atorvastatin 122 1428 2004 Bays 54 simvastatin 31 622 2004 Bays 54 placebo 2 148 2004 PREVENT IT 55 placebo 22 431 2004 PREVENT IT 55 pravastatin 13 433 2004 Durazzo 56 atorvastatin 1 50 2004 Durazzo 56 placebo 0 50 2004 Goldberg 57 placebo 2 93 2004 Goldberg 57 simvastatin 7 349 2004 ALLIANCE 58 atorvastatin 75 1217 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 53 2004 ALLIANCE 58 placebo 3 1225 2004 PCS 59 pravastatin 5 54 2004 PCS 59 placebo 0 66 2004 REVERSAL 60 pravastatin 22 327 2004 REVERSAL 60 atorvastatin 21 327 2004 DISCOVERY 61 rosuvastatin 24 686 2004 DISCOVERY 61 atorvastatin 9 338 2004 Schwatrz 62 rosuvastatin 12 255 2004 Schwatrz 62 atorvastatin 6 128 2004 Brown 63 rosuvastatin 22 239 2004 Brown 63 pravastatin 11 118 2004 Brown 63 simvastatin 9 120 2005 BELLES 64 atorvastatin 43 305 2005 BELLES 64 pravastatin 21 309 2005 DISCOVERY-Penta 65 rosuvastatin 17 358 2005 DISCOVERY-Penta 65 atorvastatin 7 383 2005 IDEAL 66 simvastatin 186 4449 2005 IDEAL 66 atorvastatin 426 4439 2005 CORALL 67 rosuvastatin 9 131 2005 CORALL 67 atorvastatin 11 132 2005 URANUS 68 rosuvastatin 3 232 2005 URANUS 68 atorvastatin 7 233 2005 COMETS 69 rosuvastatin 4 165 2005 COMETS 69 atorvastatin 4 157 2005 COMETS 69 placebo 3 79 2006 DISCOVERY-Alpha 70 rosuvastatin 23 555 2006 DISCOVERY-Alpha 70 atorvastatin 14 382 2006 SPARCL 71 atorvastatin 415 2365 2006 SPARCL 71 placebo 342 2366 2006 ASPEN 72 atorvastatin 33 1211 2006 ASPEN 72 placebo 38 1199 2006 PULSAR 73 rosuvastatin 14 504 2006 PULSAR 73 atorvastatin 11 492 2006 ARIES 74 rosuvastatin 13 391 2006 ARIES 74 atorvastatin 10 383 2006 STARSHIP 75 rosuvastatin 11 357 2006 STARSHIP 75 atorvastatin 5 339 2006 MERCURY II 76 rosuvastatin 15 392 2006 MERCURY II 76 atorvastatin 19 798 2006 MERCURY II 76 simvastatin 25 803 2007 METEOR 77 rosuvastatin 79 702 2007 METEOR 77 placebo 22 282 2007 SAGE 78 atorvastatin 48 446 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 54 2007 SAGE 78 pravastatin 46 445 2007 Kyeong 79 rosuvastatin 2 60 2007 Kyeong 79 atorvastatin 3 57 2007 ASTRONOMER 80 rosuvastatin 25 134 2007 ASTRONOMER 80 placebo 26 135 2007 CORONA 81 placebo 302 2497 2007 CORONA 81 rosuvastatin 241 2514 2007 Lewis 82 pravastatin 11 163 2007 Lewis 82 placebo 16 163 2007 ANDROMEDA 83 rosuvastatin 15 248 2007 ANDROMEDA 83 atorvastatin 13 246 2007 POLARIS 84 rosuvastatin 22 432 2007 POLARIS 84 atorvastatin 27 439 2007 DISCOVERY-Asia 85 rosuvastatin 21 950 2007 DISCOVERY-Asia 85 atorvastatin 10 472 2007 SOLAR 86 rosuvastatin 15 542 2007 SOLAR 86 atorvastatin 20 544 2007 SOLAR 86 simvastatin 20 546 2007 IRIS 87 rosuvastatin 14 371 2007 IRIS 87 atorvastatin 7 369 2008 GISSI-HF 88 rosuvastatin 104 2285 2008 GISSI-HF 88 placebo 91 2289 2008 ECLIPSE 89 rosuvastatin 41 522 2008 ECLIPSE 89 atorvastatin 36 514 2008 SUBARU 90 atorvastatin 0 213 2008 SUBARU 90 rosuvastatin 8 214 2008 DISCOVERY-Beta 91 rosuvastatin 24 334 2008 DISCOVERY-Beta 91 simvastatin 7 170 2008 Sdringola 92 placebo 3 73 2008 Sdringola 92 atorvastatin 1 72 2009 SPACE ROCKET 93 rosuvastatin 20 633 2009 SPACE ROCKET 93 simvastatin 9 630 2009 Ose 94 simvastatin 4 219 2009 Ose 94 pitavastatin 21 638 2010 CENTAURUS 95 rosuvastatin 15 437 2010 CENTAURUS 95 atorvastatin 17 450 2010 Acala 96 placebo 0 70 2010 Acala 96 pravastatin 2 61 2011 SATURN 97 atorvastatin 48 519 2011 SATURN 97 rosuvastatin 45 520 2011 Eriksson 98 pitavastatin 9 236 2011 Eriksson 98 simvastatin 6 119 2011 Gumprecht 99 pitavastatin 8 275 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 55 2011 Gumprecht 99 atorvastatin 6 137 2011 PATROL 100 atorvastatin 13 101 2011 PATROL 100 rosuvastatin 10 100 2011 PATROL 100 pitavastatin 12 101 2012 LUNAR 101 rosuvastatin 26 499 2012 LUNAR 101 atorvastatin 25 257 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 56 Table S5. NMA results from the network of randomised controlled trials comparing adverse effects of statins. Odds ratios and their 95% confidence intervals are presented. Odds ratios less than 1 favor the treatment specified in the row. Atorvastatin 0.894 (0.637, 1.255) 1.196 (0.868, 1.647) 1.127 (0.637, 1.994) 1.073 (0.890, 1.294) 1.418 (1.126, 1.785) 1.007 (0.848, 1.196) 1.297 (1.065, 1.580) 1.119 (0.797, 1.571) Fluvastatin 1.338 (0.915, 1.956) 1.261 (0.655, 2.426) 1.201 (0.879, 1.640) 1.586 (1.109, 2.268) 1.127 (0.787, 1.612) 1.451 (1.011, 2.083) 0.836 (0.607, 1.152) 0.747 (0.511, 1.093) Lovastatin 0.942 (0.494, 1.797) 0.897 (0.668, 1.206) 1.185 (0.849, 1.655) 0.842 (0.599, 1.184) 1.085 (0.768, 1.532) 0.887 (0.501, 1.570) 0.793 (0.412, 1.526) 1.061 (0.557, 2.023) Pitavastatin 0.952 (0.528, 1.718) 1.258 (0.687, 2.305) 0.893 (0.499, 1.598) 1.151 (0.648, 2.043) 0.932 (0.773, 1.123) 0.833 (0.610, 1.138) 1.114 (0.829, 1.498) 1.050 (0.582, 1.895) Placebo 1.321 (1.070, 1.632) 0.938 (0.759, 1.160) 1.209 (0.960, 1.522) 0.705 (0.560, 0.888) 0.630 (0.441, 0.902) 0.844 (0.604, 1.178) 0.795 (0.434, 1.457) 0.757 (0.613, 0.935) Pravastatin 0.710 (0.549, 0.918) 0.915 (0.702, 1.193) 0.993 (0.836, 1.179) 0.888 (0.620, 1.270) 1.188 (0.844, 1.671) 1.119 (0.626, 2.002) 1.066 (0.862, 1.317) 1.408 (1.089, 1.821) Rosuvastatin 1.288 (1.024, 1.621) 0.771 (0.633, 0.939) 0.689 (0.480, 0.989) 0.922 (0.653, 1.302) 0.869 (0.489, 1.542) 0.827 (0.657, 1.041) 1.093 (0.839, 1.424) 0.776 (0.617, 0.977) Simvastatin .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; 57 Table S6. Results from SIDE splitting for three network comparisons of the network of statins. OR: odds ratio. SIDE: separate indirect from direct approach. Comparison Direct OR Indirect OR Ratio of ORs z-value p-value Pravastatin versus rosuvastatin 0.98 0.67 1.47 1.06 0.29 Pravastatin versus simvastatin 0.84 0.95 0.89 -0.42 0.67 Rosuvastatin versus simvastatin 1.23 1.32 0.93 -0.27 0.78 .CC-BY-NC-ND 4.0 International licenseIt is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint. http://dx.doi.org/10.1101/597047doi: bioRxiv preprint first posted online Apr. 5, 2019; Citations (22)References (48)... For outcomes analyzed by pairwise meta-analysis or no meta-analysis, we will follow current GRADE guidance [23,34,[65][66][67]. For findings from NMA, we will be guided by the CINeMa approach and use CINeMA software for some assessments, which is based on the GRADE framework, although has conceptual and semantic differences [68]. The assessment covers six domains: within-study bias, across-studies bias (i.e., publication and other reporting biases), indirectness, imprecision, heterogeneity (i.e., variation between studies within a comparison), and incoherence (i.e., variation between direct and indirect sources of evidence across comparisons). ...... Each outcome starts at high certainty and is rated down for concerns. The six CINeMA domains are interconnected and should be considered jointly rather than in isolation [68]; if two concerns are highly related, we will not rate down twice. ...... Suspected bias entails (i) failure to include unpublished data, (ii) meta-analysis is based on a small number of positive early studies, (iii) the comparison has been funded primarily by industry-funded trials, or (iv) there is existing evidence of reporting bias. A judgment of undetected bias arises from (i) inclusion of unpublished studies with similar findings to those published, (ii) protocols and clinical trial registries are available for many trials and important discrepancies are not found, and (iii) the effects from small studies do not differ from those from large studies [68]. Although our inclusion of gray literature and many studies, as well as the non-pharmacologic topic, would suggest no suspicion of bias, we expect [29] a large portion of the studies to have concerns about selective reporting (e.g., missing outcomes). ...Fall prevention interventions for older community-dwelling adults: systematic reviews on benefits, harms, and patient values and preferencesArticleFull-text availableJan 2021 Jennifer Pillay John J Riva Laure TessierLisa HartlingBackground An estimated 20–30% of community-dwelling Canadian adults aged 65 years or older experience one or more falls each year. Fall-related injuries are a leading cause of hospitalization and can lead to functional independence. Many fall prevention interventions, often based on modifiable risk factors, have been studied. Apart from the magnitude of the benefits and harms from different interventions, the preferences of older adults for different interventions as well as the relative importance they place on the different potential outcomes may influence recommendations by guideline panels. These reviews on benefits and harms of interventions, and on patient values and preferences, will inform the Canadian Task Force on Preventive Health Care to develop recommendations on fall prevention for primary care providers. Methods To review the benefits and harms of fall prevention interventions, we will update a previous systematic review of randomized controlled trials with adaptations to modify the classification of interventions and narrow the scope to community-dwelling older adults and primary-care relevant interventions. Four databases (MEDLINE, Embase, Cochrane Central Register of Controlled Trials, Ageline), reference lists, trial registries, and relevant websites will be searched, using limits for randomized trials and date (2016 onwards). We will classify interventions according to the Prevention of Falls Network Europe (ProFANE) Group’s taxonomy. Outcomes include fallers, falls, injurious falls, fractures, hip fractures, institutionalization, health-related quality of life, functional status, and intervention-related adverse effects. For studies not included in the previous review, screening, study selection, data extraction on outcomes, and risk of bias assessments will be independently undertaken by two reviewers with consensus used for final decisions. Where quantitative analysis is suitable, network or pairwise meta-analysis will be conducted using a frequentist approach in Stata. Assessment of the transitivity and coherence of the network meta-analyses will be undertaken. For the reviews on patient preferences and outcome valuation (relative importance of outcomes), we will perform de novo reviews with searches in three databases (MEDLINE, PsycInfo, and CINAHL) and reference lists for cross-sectional, longitudinal quantitative, or qualitative studies published from 2000. Selection, data extraction, and risk of bias assessments suitable for each study design will be performed in duplicate. The analysis will be guided by a narrative synthesis approach, which may include meta-analysis for health-state utilities. We will use the CINeMa approach to a rate the certainty of the evidence for outcomes on intervention effects analyzed using network meta-analysis and the GRADE approach for all other outcomes. Discussion We will describe the flow of literature and characteristics of all studies and present results of all analyses and summary of finding tables. We will compare our findings to others and discuss the limitations of the reviews and the available literature. Systematic review registration This protocol has not been registered.ViewShow abstract... Although we also performed NMAs for ACR20, ACR50, PASI75, and PASI90, based on the current expectations on the efficacy of new biologic treatments and on the confidence in the results, we decided to present the efficacy of the different biologic therapy using ACR70 ( Table 4) and PASI100 (Table 8), the most challenging outcomes. The confidence rating on direct and indirect estimates was calculated using CINeMA to improve the transparency and limit the subjectivity of the process (90)(91)(92). Comparisons with a high confidence rating, based on the CINeMA evaluation (91), are represented in bold. The level of confidence of the other comparisons is either low or very low, and consequently, the surface under the cumulative rating (SUCRA) will result in misleading inferences (90,93). ...... Comparisons with a high confidence rating, based on the CINeMA evaluation (91), are represented in bold. The level of confidence of the other comparisons is either low or very low, and consequently, the surface under the cumulative rating (SUCRA) will result in misleading inferences (90,93). Thus, a SUCRA was not done and, therefore, it was impossible to rank the available biologic treatments. ...A Systematic Review With Network Meta-Analysis of the Available Biologic Therapies for Psoriatic Disease DomainsArticleFull-text availableJan 2021 Tiago Torres Anabela BarcelosPaulo Filipe João FonsecaIntroduction: Several new treatments have been developed for psoriatic disease, an inflammatory condition that involves skin and joints. Notwithstanding, few studies have made direct comparisons between treatments and therefore it is difficult to select the ideal treatment for an individual patient. The aim of this systematic review with network meta-analysis (NMA) was to analyze available and approved biologic therapies for each domain of psoriatic disease: skin, peripheral arthritis, axial arthritis, enthesitis, dactylitis, and nail involvement. Methods: Data from randomized clinical trials (RCTs) were included. A systematic review was performed using the MEDLINE database (July 2020) using PICO criteria. Bayesian NMA was conducted to compare the clinical efficacy of biological therapy in terms of the American College of Rheumatology criteria (ACR, 24 weeks) and Psoriasis Area and Severity Index (PASI, 10–16 weeks). Results: Fifty-four RCTs were included in the systematic review. Due to the design of the RCTs, namely, outcomes and time points, network meta-analysis was performed for skin and peripheral arthritis domains. For the skin domain, 30 studies reporting PASI100 were included. The peripheral arthritis domain was analyzed through ACR70 in 12 studies. From the therapies approved for both domains, secukinumab and ixekizumab were the ones with the highest probability of reaching the proposed outcomes. There is a lack of outcome uniformization in the dactylitis, enthesitis, and nail domains, and therefore, an objective comparison of the studies was not feasible. Nevertheless, secukinumab was the treatment with the best compromise between the number of studies in each domain and the results obtained in the different outcomes. Conclusion: Secukinumab and ixekizumab were the treatments with the highest probability of reaching both PASI100 and ACR70 outcomes. Due to the lack of a standard evaluation of outcomes of the other psoriatic disease domains, a network meta-analysis for all the domains was not possible to perform.ViewShow abstract... 23,24 The certainty of evidence produced by the synthesis for each outcome was evaluated using the framework described by Salanti and colleagues 25 and implemented using the CINeMA (Confidence in Network Meta-Analysis) web application which allows confidence in the results to be graded as high, moderate, low, and very low (appendix pp 238-67). 26 For the primary outcome we examined the confidence of evidence of all comparisons. ...Comparative Efficacy and Tolerability of 32 Oral Antipsychotics for the Acute Treatment of Adults With Multi-Episode Schizophrenia: A Systematic Review and Network Meta-AnalysisArticleOct 2020 Maximilian Huhn Adriani Nikolakopoulou Johannes Schneider-Thoma Stefan Leucht(Reprinted with permission from the Lancet 2019; 394: 939-51).ViewShow abstract... to group outcomes by their type, e.g. efficacy/acceptability/safety; to use different coloring schemes for different classes of outcomes; to color the cells after using the CINEMA approach 22 to assess the quality of the evidence (online tool available in https://cinema.ispm.unibe.ch/). We hereby did not pursue such extensions, aiming to keep the plot as simple as possible. ...The Kilim plot: A tool for visualizing network meta‐analysis results for multiple outcomesArticleFull-text availableJun 2020Michael SeoToshi A Furukawa Areti Angeliki Veroniki Orestis EfthimiouNetwork meta‐analysis (NMA) can be used to compare multiple competing treatments for the same disease. In practice, usually a range of outcomes are of interest. As the number of outcomes increases, summarizing results from multiple NMAs becomes a non‐trivial task, especially for larger networks. Moreover, NMAs provide results in terms of relative effect measures that can be difficult to interpret and apply in every‐day clinical practice, such as the odds ratios. In this paper, we aim to facilitate the clinical decision‐making process by proposing a new graphical tool, the Kilim plot, for presenting results from NMA on multiple outcomes. Our plot compactly summarizes results on all treatments and all outcomes; it provides information regarding the strength of the statistical evidence of treatment effects, while it illustrates absolute, rather than relative, effects of interventions. Moreover, it can be easily modified to include considerations regarding clinically important effects. To showcase our method, we use data from a network of studies in antidepressants. All analyses are performed in R and we provide the source code needed to produce the Kilim plot, as well as an interactive web application.ViewShow abstractComparative efficacy and acceptability of psychosocial treatments for disruptive behaviour disorders in children and adolescents: study protocol for a systematic review and network meta-analysisArticleFull-text availableJun 2021 Lin ZhangZhihong RenXueyao MaFenghui YuanIntroduction Disruptive behaviour disorders are common among children and adolescents, with negative impacts on the youths, their families and society. Although multiple psychosocial treatments are effective in decreasing the symptoms of disruptive behaviour disorders, comprehensive evidence regarding the comparative efficacy and acceptability between these treatments is still lacking. Therefore, we propose a systematic review and network meta-analysis, integrating both direct and indirect comparisons to obtain a hierarchy of treatment efficacy and acceptability. Methods and analysis The present protocol will be reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols. Ten databases, including Web of Science, PubMed, PsycINFO, MEDLINE, APA PsycArticles, Psychology and Behavioral Sciences Collection, OpenDissertations, The Cochrane Library, Embase and CINAHL, will be searched from inception for randomised controlled trials of psychosocial treatments for children and adolescents with disruptive behaviour disorders, without restrictions on language, publication year and status. The primary outcomes will be efficacy at post-treatment (severity of disruptive behaviour disorders at post-treatment) and acceptability (dropout rate for any reason) of psychosocial treatments. The secondary outcomes will involve efficacy at follow-up, severity of internalising problems and improvement of social functioning. Two authors will independently conduct the study selection and data extraction, assess the risk of bias using the revised Cochrane Collaboration’s Risk of Bias tool and evaluate the quality of the evidence using the Grading of Recommendations Assessment, Development and Evaluation framework to network meta-analysis. We will perform Bayesian network meta-analyses with a random effects model. Subgroup and sensitivity analyses will be performed to evaluate the robustness of the findings. Ethics and dissemination The research does not require ethical approval. Results are planned to be published in journals or presented at conferences. The network meta-analysis will provide information on a hierarchy of treatment efficacy and acceptability and help make a clinical treatment choice. PROSPERO registration number CRD42020197448.ViewShow abstractThe effect of diet, exercise, and lifestyle intervention on childhood obesity: A network meta-analysisArticleNov 2020CLIN NUTRJi-Hyun Bae Hyorim LeeBackground aims Trials investigating the efficacy of different interventions for overweight children are limited and controversial. Therefore, the aim of this study is to perform a network meta-analysis on the efficacy of various interventions for children with obesity (an average age of 6–12 years old). Methods We obtained the data of trials reporting pre-post obesity relevant outcomes (e.g. BMI, BMI z-score, percent body fat, or percent overweight) from the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE (Ovid), PubMed, and Web of Science databases (completed before February 25, 2019) and included at least one pair of direct comparison groups. The mean difference of outcomes and their associated 95% CI were used to determine the efficacy. The P-score was calculated to illustrate the rank probability of various treatments for different outcomes using a network meta-analysis. Our meta-analysis included 24 studies that evaluated the interventions for childhood obesity. Results All 24 trials had no high risk of bias. Interventions such as exercise without parents (E w/o P); diet with parents (D w/P); and diet, exercise, and lifestyle with parents (D+E+L w/P) were significantly effective for children with obesity when compared with no intervention. Conclusions E w/o P exhibited the highest P-score, with the D w/P and D+E+L w/P ranks having P-scores of 0.7486 and 0.5464, respectively. Moreover, the results indicate that E w/o P, D w/P, and D+E+L w/P were significantly effective treatments for children with obesity when compared with no intervention.ViewShow abstractInterventions for the treatment of COVID-19: a living network meta-analysisArticleNov 2020COCHRANE DB SYST REV Isabelle Boutron Anna ChaimaniDeclan Devane Philippe RavaudViewInterventions for the prevention and treatment of COVID-19: a living mapping of research and living network meta-analysisArticleNov 2020COCHRANE DB SYST REV Isabelle Boutron Anna ChaimaniDeclan Devane Philippe RavaudObjectives: This is a protocol for a Cochrane Review (intervention). The objectives are as follows:. To provide a living mapping of registered randomized trials, and to assess and rank where appropriate the relative effects of interventions for the prevention and treatment of COVID-19. Our approach has been described in Boutron 2020 and Nguyen 2020. As part of the methodological process of living systematic reviews, we will continuously (i.e. every working day) collect and critically appraise results from all eligible RCTs addressing specific clinical outcomes related to COVID-19. We will synthesize the available study results using pairwise meta-analyses and when possible and appropriate, network meta-analyses (NMAs). The interventions and the research questions considered will evolve over time and will be guided by end-users’ needs. We will report our updated evidence synthesis every week, and will make all results available on a website: www.covid-nma.com. In addition, we will update this Cochrane Review at least once every six months, or as soon as the certainty of evidence (assessed with the GRADE methodology) changes. We will wait until the accumulating evidence changes one or more of the following aspects of the review, before incorporating it and re-publishing the Cochrane Review:. the findings of one or more critical outcomes; the credibility (e.g. GRADE rating) of one or more critical outcomes; new settings, population, interventions, comparisons or outcomes studied; or new serious adverse events. We will develop a steering committee of epidemiologists, methodologists, statisticians and clinicians with content expertise. This committee will meet regularly and discuss the conduct of the project, difficulties encountered and possible changes in the protocol according to new knowledge becoming available for the disease. Changes to the protocol could include, for example, changes in the search strategy, eligibility criteria (e.g. study design), research questions for the pairwise meta-analyses, or outcomes. The steering committee will systematically discuss and achieve consensus on the changes to the protocol. The process will also evolve over time according to new knowledge available regarding COVID-19. Copyright © 2020 The Cochrane Collaboration. Published by John Wiley Sons, Ltd.ViewShow abstractErector spinae plane block for postoperative painArticleOct 2020COCHRANE DB SYST REV Alexander Schnabel Stephanie Weibel Michael MeißnerEsther Pogatzki-ZahnViewProtocol for a systematic review and network meta-analysis of randomised controlled trials examining the effectiveness of early parenting interventions in preventing internalising problems in children and adolescentsArticleFull-text availableOct 2020 Ilaria CostantiniElise PaulDeborah M. Caldwell Rebecca M PearsonBackground Internalising problems, such as depression and anxiety, are common and represent an important economical and societal burden. The effectiveness of parenting interventions in reducing the risk of internalising problems in children and adolescents has not yet been summarised. The aims of this review are to assess the effectiveness of parenting interventions in the primary, secondary and tertiary prevention of internalising problems in children and adolescents and to determine which intervention components and which intervention aspects are most effective for reducing the risk of internalising problems in children and adolescents. Methods Electronic searches in OVID SP versions of MEDLINE, EMBASE and PsycINFO; Cochrane Central Register of Controlled Trials; EBSCO version of ERIC and ClinicalTrials.gov have been performed to identify randomised controlled trials or quasi-randomised controlled trials of parenting interventions. At least two independent researchers will assess studies for inclusion and extract data from each paper. The risk of bias assessment will be conducted independently by two reviewers using the Cochrane Collaboration’s Risk of Bias Assessment Tool. Statistical heterogeneity is anticipated given potential variation in participant characteristics, intervention type and mode of delivery, and outcome measures. Random effects models, assuming a common between-study variability, will be used to account for statistical heterogeneity. Results will be analysed using a network meta-analysis (NMA). If appropriate, we will also conduct a component-level NMA, where the ‘active ingredients’ of interventions are modelled using a network meta-regression approach. Discussion Preventing and reducing internalising problems could have major beneficial effects at the economic and societal level. Informing policy makers on the effectiveness of parenting interventions and on which intervention’s component is driving the effect is important for the development of treatment strategies. Systematic review registration International Prospective Register for Systematic Reviews (PROSPERO) number CRD42020172251ViewShow abstractShow moreEstimating the contribution of studies in network meta-analysis: paths, flows and streamsArticleFull-text availableMay 2018 Thodoris Papakonstantinou Adriani NikolakopoulouGerta Rücker Georgia SalantiIn network meta-analysis, it is important to assess the influence of the limitations or other characteristics of individual studies on the estimates obtained from the network. The percentage contribution matrix, which shows how much each direct treatment effect contributes to each treatment effect estimate from network meta-analysis, is crucial in this context. We use ideas from graph theory to derive the percentage that is contributed by each direct treatment effect. We start with the projection matrix in a two-step network meta-analysis model, called the matrix, which is analogous to the hat matrix in a linear regression model. We develop a method to translate entries to percentage contributions based on the observation that the rows of can be interpreted as flow networks, where a stream is defined as the composition of a path and its associated flow. We present an algorithm that identifies the flow of evidence in each path and decomposes it into direct comparisons. To illustrate the methodology, we use two published networks of interventions. The first compares no treatment, quinolone antibiotics, non-quinolone antibiotics and antiseptics for underlying eardrum perforations and the second compares 14 antimanic drugs. We believe that this approach is a useful and novel addition to network meta-analysis methodology, which allows the consistent derivation of the percentage contributions of direct evidence from individual studies to network treatment effects.ViewShow abstractOutcomes of non-invasive diagnostic modalities for the detection of coronary artery disease: network meta-analysis of diagnostic randomised controlled trialsArticleFull-text availableFeb 2018Br Med JGeorge CM Siontis Dimitris MavridisJohn P GreenwoodStephan WindeckerObjective To evaluate differences in downstream testing, coronary revascularisation, and clinical outcomes following non-invasive diagnostic modalities used to detect coronary artery disease. Design Systematic review and network meta-analysis. Data sources Medline, Medline in process, Embase, Cochrane Library for clinical trials, PubMed, Web of Science, SCOPUS, WHO International Clinical Trials Registry Platform, and Clinicaltrials.gov. Eligibility criteria for selecting studies Diagnostic randomised controlled trials comparing non-invasive diagnostic modalities in patients presenting with symptoms suggestive of low risk acute coronary syndrome or stable coronary artery disease. Data synthesis A random effects network meta-analysis synthesised available evidence from trials evaluating the effect of non-invasive diagnostic modalities on downstream testing and patient oriented outcomes in patients with suspected coronary artery disease. Modalities included exercise electrocardiograms, stress echocardiography, single photon emission computed tomography-myocardial perfusion imaging, real time myocardial contrast echocardiography, coronary computed tomographic angiography, and cardiovascular magnetic resonance. Unpublished outcome data were obtained from 11 trials. Results 18 trials of patients with low risk acute coronary syndrome (n=11 329) and 12 trials of those with suspected stable coronary artery disease (n=22 062) were included. Among patients with low risk acute coronary syndrome, stress echocardiography, cardiovascular magnetic resonance, and exercise electrocardiograms resulted in fewer invasive referrals for coronary angiography than coronary computed tomographic angiography (odds ratio 0.28 (95% confidence interval 0.14 to 0.57), 0.32 (0.15 to 0.71), and 0.53 (0.28 to 1.00), respectively). There was no effect on the subsequent risk of myocardial infarction, but estimates were imprecise. Heterogeneity and inconsistency were low. In patients with suspected stable coronary artery disease, an initial diagnostic strategy of stress echocardiography or single photon emission computed tomography-myocardial perfusion imaging resulted in fewer downstream tests than coronary computed tomographic angiography (0.24 (0.08 to 0.74) and 0.57 (0.37 to 0.87), respectively). However, exercise electrocardiograms yielded the highest downstream testing rate. Estimates for death and myocardial infarction were imprecise without clear discrimination between strategies. Conclusions For patients with low risk acute coronary syndrome, an initial diagnostic strategy of stress echocardiography or cardiovascular magnetic resonance is associated with fewer referrals for invasive coronary angiography and revascularisation procedures than non-invasive anatomical testing, without apparent impact on the future risk of myocardial infarction. For suspected stable coronary artery disease, there was no clear discrimination between diagnostic strategies regarding the subsequent need for invasive coronary angiography, and differences in the risk of myocardial infarction cannot be ruled out. Systematic review registration PROSPERO registry no CRD42016049442.ViewShow abstractCharacteristics and knowledge synthesis approach for 456 network meta-analyses: A scoping reviewArticleFull-text availableJan 2017BMC MED Wasifa Zarin Areti Angeliki Veroniki Vera Nincic Andrea TriccoBackground Network meta-analysis (NMA) has become a popular method to compare more than two treatments. This scoping review aimed to explore the characteristics and methodological quality of knowledge synthesis approaches underlying the NMA process. We also aimed to assess the statistical methods applied using the Analysis subdomain of the ISPOR checklist. Methods Comprehensive literature searches were conducted in MEDLINE, PubMed, EMBASE, and Cochrane Database of Systematic Reviews from inception until April 14, 2015. References of relevant reviews were scanned. Eligible studies compared at least four different interventions from randomised controlled trials with an appropriate NMA approach. Two reviewers independently performed study selection and data abstraction of included articles. All discrepancies between reviewers were resolved by a third reviewer. Data analysis involved quantitative (frequencies) and qualitative (content analysis) methods. Quality was evaluated using the AMSTAR tool for the conduct of knowledge synthesis and the ISPOR tool for statistical analysis. Results After screening 3538 citations and 877 full-text papers, 456 NMAs were included. These were published between 1997 and 2015, with 95% published after 2006. Most were conducted in Europe (51%) or North America (31%), and approximately one-third reported public sources of funding. Overall, 84% searched two or more electronic databases, 62% searched for grey literature, 58% performed duplicate study selection and data abstraction (independently), and 62% assessed risk of bias. Seventy-eight (17%) NMAs relied on previously conducted systematic reviews to obtain studies for inclusion in their NMA. Based on the AMSTAR tool, almost half of the NMAs incorporated quality appraisal results to formulate conclusions, 36% assessed publication bias, and 16% reported the source of funding. Based on the ISPOR tool, half of the NMAs did not report if an assessment for consistency was conducted or whether they accounted for inconsistency when present. Only 13% reported heterogeneity assumptions for the random-effects model. Conclusions The knowledge synthesis methods and analytical process for NMAs are poorly reported and need improvement. Electronic supplementary material The online version of this article (doi:10.1186/s12916-016-0764-6) contains supplementary material, which is available to authorized users.ViewShow abstractUse of network meta-analysis in clinical guidelinesArticleFull-text availableOct 2016B WORLD HEALTH ORGAN Steve KantersNathan Ford Eric DruytsNick BansbackViewA threshold analysis assessed the credibility of conclusions from network meta-analysisArticleFull-text availableJul 2016J CLIN EPIDEMIOL Deborah M CaldwellAE Ades Sofia Dias Nicky J WeltonObjective: To assess the reliability of treatment recommendations based on network meta-analysis (NMA) STUDY DESIGN: We consider evidence in an NMA to be potentially biased. Taking each pair-wise contrast in turn we use a structured series of threshold analyses to ask: (a) How large would the bias in this evidence-base have to be before it changed our decision? and (b) If the decision changed, what is the new recommendation? We illustrate the method via two NMAs in which a GRADE assessment for NMAs has been implemented: weight-loss and osteoporosis. Results: Four of the weight-loss NMA estimates were assessed as low and 6 as moderate quality by GRADE; for osteoporosis 6 were low , 9 moderate and 1 high . The threshold analysis suggests plausible bias in 3 of 10 estimates in the weight-loss network could have changed the treatment recommendation. For osteoporosis plausible bias in 6 of 16 estimates could change the recommendation. There was no relation between plausible bias changing a treatment recommendation and the original GRADE assessments. Conclusions: Reliability judgements on individual NMA contrasts do not help decision makers understand whether a treatment recommendation is reliable. Threshold analysis reveals whether the final recommendation is robust against plausible degrees of bias in the data.ViewShow abstractRoB 2: A revised tool for assessing risk of bias in randomised trialsArticleAug 2019Br Med J Jonathan A C SterneJelena Savović Matthew J PageJulian P T HigginsAssessment of risk of bias is regarded as an essential component of a systematic review on the effects of an intervention. The most commonly used tool for randomised trials is the Cochrane risk-of-bias tool. We updated the tool to respond to developments in understanding how bias arises in randomised trials, and to address user feedback on and limitations of the original tool.ViewShow abstractComparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta-analysisArticleFeb 2018LANCET Andrea CiprianiToshi A Furukawa Georgia Salanti John R GeddesBackground: Major depressive disorder is one of the most common, burdensome, and costly psychiatric disorders worldwide in adults. Pharmacological and non-pharmacological treatments are available; however, because of inadequate resources, antidepressants are used more frequently than psychological interventions. Prescription of these agents should be informed by the best available evidence. Therefore, we aimed to update and expand our previous work to compare and rank antidepressants for the acute treatment of adults with unipolar major depressive disorder. Methods: We did a systematic review and network meta-analysis. We searched Cochrane Central Register of Controlled Trials, CINAHL, Embase, LILACS database, MEDLINE, MEDLINE In-Process, PsycINFO, the websites of regulatory agencies, and international registers for published and unpublished, double-blind, randomised controlled trials from their inception to Jan 8, 2016. We included placebo-controlled and head-to-head trials of 21 antidepressants used for the acute treatment of adults (≥18 years old and of both sexes) with major depressive disorder diagnosed according to standard operationalised criteria. We excluded quasi-randomised trials and trials that were incomplete or included 20% or more of participants with bipolar disorder, psychotic depression, or treatment-resistant depression; or patients with a serious concomitant medical illness. We extracted data following a predefined hierarchy. In network meta-analysis, we used group-level data. We assessed the studies risk of bias in accordance to the Cochrane Handbook for Systematic Reviews of Interventions, and certainty of evidence using the Grading of Recommendations Assessment, Development and Evaluation framework. Primary outcomes were efficacy (response rate) and acceptability (treatment discontinuations due to any cause). We estimated summary odds ratios (ORs) using pairwise and network meta-analysis with random effects. This study is registered with PROSPERO, number CRD42012002291. Findings: We identified 28 552 citations and of these included 522 trials comprising 116 477 participants. In terms of efficacy, all antidepressants were more effective than placebo, with ORs ranging between 2·13 (95% credible interval [CrI] 1·89-2·41) for amitriptyline and 1·37 (1·16-1·63) for reboxetine. For acceptability, only agomelatine (OR 0·84, 95% CrI 0·72-0·97) and fluoxetine (0·88, 0·80-0·96) were associated with fewer dropouts than placebo, whereas clomipramine was worse than placebo (1·30, 1·01-1·68). When all trials were considered, differences in ORs between antidepressants ranged from 1·15 to 1·55 for efficacy and from 0·64 to 0·83 for acceptability, with wide CrIs on most of the comparative analyses. In head-to-head studies, agomelatine, amitriptyline, escitalopram, mirtazapine, paroxetine, venlafaxine, and vortioxetine were more effective than other antidepressants (range of ORs 1·19-1·96), whereas fluoxetine, fluvoxamine, reboxetine, and trazodone were the least efficacious drugs (0·51-0·84). For acceptability, agomelatine, citalopram, escitalopram, fluoxetine, sertraline, and vortioxetine were more tolerable than other antidepressants (range of ORs 0·43-0·77), whereas amitriptyline, clomipramine, duloxetine, fluvoxamine, reboxetine, trazodone, and venlafaxine had the highest dropout rates (1·30-2·32). 46 (9%) of 522 trials were rated as high risk of bias, 380 (73%) trials as moderate, and 96 (18%) as low; and the certainty of evidence was moderate to very low. Interpretation: All antidepressants were more efficacious than placebo in adults with major depressive disorder. Smaller differences between active drugs were found when placebo-controlled trials were included in the analysis, whereas there was more variability in efficacy and acceptability in head-to-head trials. These results should serve evidence-based practice and inform patients, physicians, guideline developers, and policy makers on the relative merits of the different antidepressants. Funding: National Institute for Health Research Oxford Health Biomedical Research Centre and the Japan Society for the Promotion of Science.ViewShow abstractAdvances in the GRADE approach to rate the certainty in estimates from a network meta-analysisArticleOct 2017J CLIN EPIDEMIOL Romina Brignardello-PetersenAshley BonnerPaul Alexander Gordon H GuyattThis article describes conceptual advances of the Grading of Recommendations Assessments, Development, and Evaluation (GRADE) working group guidance to evaluate certainty of evidence (confidence in evidence, quality of evidence) from network meta-analysis (NMA). Application of the original GRADE guidance, published in 2014, in a number of NMAs has resulted in advances that strengthen its conceptual basis and make the process more efficient. This guidance will be useful for systematic reviewer authors who aim to assess the certainty of all pairwise comparisons from an NMA and who are familiar with the basic concepts of NMA and the traditional GRADE approach for pairwise meta-analysis. Two principles of the original GRADE NMA guidance are that we need to rate the certainty of the evidence for each pairwise comparison within a network separately, and that in doing so we need to consider both the direct and indirect evidence. We present, discuss, and illustrate four conceptual advances: 1) Consideration of imprecision is not necessary when rating the direct and indirect estimates to inform the rating of NMA estimates, 2) There is no need to rate the indirect evidence when the certainty of the direct evidence is high and the contribution of the direct evidence to the network estimate is at least as great as that of the indirect evidence, 3) We should not trust a statistical test of global incoherence of the network to assess incoherence at the pairwise comparison level, and 4) In the presence of incoherence between direct and indirect evidence, the certainty of the evidence of each estimate can help decide which estimate to believe.ViewShow abstractBibliographic study showed improving statistical methodology of network meta-analyses published between 1999 and 2015ArticleNov 2016 Maria Petropoulou Adriani Nikolakopoulou Areti Angeliki Veroniki Georgia SalantiObjective: To assess the characteristics and core statistical methodology specific to network meta-analyses (NMAs) in clinical research articles. Study design and setting: We searched Medline, Embase and the Cochrane Database of Systematic Reviews from inception until April 14, 2015 for NMAs of randomized controlled trials (RCTs) including at least four different interventions. Two reviewers independently screened potential studies, while data abstraction was performed by a single reviewer and verified by a second. Results: A total of 456 NMAs, which included a median (interquartile range) of 21 (13 to 40) studies and 7 (5 to 9) treatment nodes were assessed. A total of 125 NMAs (27%) were star networks; this proportion declined from 100% in 2005 to 19% in 2015 (p=0.01 by test of trend). An increasing number of NMAs discussed transitivity or inconsistency (0% in 2005, 86% in 2015, p 0.01) and 150 (45 %) and used appropriate methods to test for inconsistency (14% in 2006, 74% in 2015, p 0.01). Heterogeneity was explored in 256 NMAs (56%), with no change over time (p=0.10). All pairwise effects were reported in 234 NMAs (51%), with some increase over time (p=0.02). The hierarchy of treatments was presented in 195 NMAs (43%), the probability of being best was most commonly reported (137 NMAs, 70%) but use of SUCRA (surface under the cumulative ranking curves) increased steeply (0% in 2005, 33% in 2015, p 0.01). Conclusion: Many NMAs published in the medical literature have significant limitations in both the conduct and reporting of the statistical analysis and numerical results. The situation has however improved in recent years, in particular with respect to the evaluation of the underlying assumptions, but considerable room for further improvements remains.ViewShow abstractThe Cochrane Collaboration s tool for assessing risk of bias in randomized trialsArticleOct 2011Julian P T Higgins Douglas AltmanPeter C Gøtzsche Jonathan A C SterneViewShow moreRecommendationsDiscover moreProjectCINeMA Thodoris Papakonstantinou Adriani Nikolakopoulou Anna Chaimani[...]Julian P T HigginsCINeMA (Confidence in Network Meta-Analysis) is a web application that simplifies the evaluation of confidence in the findings from network meta-analysis. It is based on a framework described in ( 1) which considers the five GRADE domains: study limitations, indirectness, inconsistency, imprecision and publication bias. The framework combines judgments about direct evidence with their statistical contribution to network meta-analysis results, enabling evaluation of the credibility of NMA treatment effects. 1. Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JPT. Evaluating the quality of evidence from a network meta-analysis. PloS One. 2014;9(7):e99682. ... [more]View projectProjectMissing outcome data in meta-analysis models Dimitris Mavridis Georgia Salanti Anna Chaimani[...]Stefan LeuchtDevelop statistical models for missing outcome data in aggregate-data meta-analysis View projectProjectThe next step in evidence-based treatment of schizophrenia. Individualising the care for important patient subgroups Marc KrauseStefan Leucht Maximilian Huhn[...] Irene BighelliTo examine the comparative efficacy, acceptability, and tolerability of antipsychotic drugs in important patient subgroups for schizophrenia by applying a network meta-analysis approach. View projectProjectIMI GetReal Thomas P A Debray Orestis Efthimiou Georgia Salanti[...]Klea PanayidouIMI GetReal aims to show how robust new methods of real world evidence collection and synthesis could be adopted earlier in pharmaceutical R D and the healthcare decision making process. View projectArticleFull-text availableTrial Sequential Analysis in systematic reviews with meta-analysisMarch 2017 · BMC Medical Research Methodology Jørn Wetterslev Janus Christian Jakobsen Christian GluudBackground Most meta-analyses in systematic reviews, including Cochrane ones, do not have sufficient statistical power to detect or refute even large intervention effects. This is why a meta-analysis ought to be regarded as an interim analysis on its way towards a required information size. The results of the meta-analyses should relate the total number of randomised participants to the estimated ... [Show full abstract] required meta-analytic information size accounting for statistical diversity. When the number of participants and the corresponding number of trials in a meta-analysis are insufficient, the use of the traditional 95% confidence interval or the 5% statistical significance threshold will lead to too many false positive conclusions (type I errors) and too many false negative conclusions (type II errors). Methods We developed a methodology for interpreting meta-analysis results, using generally accepted, valid evidence on how to adjust thresholds for significance in randomised clinical trials when the required sample size has not been reached. ResultsThe Lan-DeMets trial sequential monitoring boundaries in Trial Sequential Analysis offer adjusted confidence intervals and restricted thresholds for statistical significance when the diversity-adjusted required information size and the corresponding number of required trials for the meta-analysis have not been reached. Trial Sequential Analysis provides a frequentistic approach to control both type I and type II errors. We define the required information size and the corresponding number of required trials in a meta-analysis and the diversity (D2) measure of heterogeneity. We explain the reasons for using Trial Sequential Analysis of meta-analysis when the actual information size fails to reach the required information size. We present examples drawn from traditional meta-analyses using unadjusted naïve 95% confidence intervals and 5% thresholds for statistical significance. Spurious conclusions in systematic reviews with traditional meta-analyses can be reduced using Trial Sequential Analysis. Several empirical studies have demonstrated that the Trial Sequential Analysis provides better control of type I errors and of type II errors than the traditional naïve meta-analysis. Conclusions Trial Sequential Analysis represents analysis of meta-analytic data, with transparent assumptions, and better control of type I and type II errors than the traditional meta-analysis using naïve unadjusted confidence intervals.View full-textArticleImatinib discontinuation in chronic myeloid leukaemia patients with undetectable BCR-ABL transcript...March 2017 · European journal of cancer (Oxford, England: 1990) Leonardo Campiotti Matteo Basilio SuterLuigina Guasti[...] Alessandro SquizzatoPurpose: Tyrosine kinase inhibitors (TKIs) are the cornerstones of treatment for patients with chronic myeloid leukaemia (CML). In recent years, several studies were conducted to evaluate the safety of TKIs discontinuation. We performed a systematic review of the literature to determine the incidence of CML relapse, to identify possible factors relapse rates?and to evaluate the long-term safety ... [Show full abstract] in CML patients with stable undetectable BCR-ABL transcript level who discontinued TKIs. Design: Studies evaluating TKIs discontinuation in CML patients with undetectable BCR-ABL transcript level were identified by electronic search of MEDLINE and EMBASE database until May 2015. Weighted mean proportion and 95% confidence intervals (CIs) of CML relapse was calculated using a fixed-effects and a random-effects model. Statistical heterogeneity was evaluated using the I(2) statistic. Results: Fifteen cohort studies, for a total of 509 patients, were included. Nine studies were at low-risk of bias. All 15 studies included only patients on imatinib. Overall weighted mean molecular relapse rate of CML was 51% (95% CI 44-58%; I(2)?=?55). Weighted mean molecular relapse rate at 6-month follow-up was 41% (95% CI 32-51%; I(2)?=?78). Eighty percent of molecular relapses occurred in the first 6 months. All 509 patients were alive at 2-year follow-up and only one patient (0.8%, 95% CI 0.2-1.8%; I(2)?=?0) has progressed to a blastic crisis. Conclusions: Our findings suggest that imatinib discontinuation is feasible for the majority of CML patients with stable undetectable BCR-ABL transcript level. Approximately 50% of patients remain therapy-free after imatinib discontinuation. Restarting TKIs therapy was followed by a very high rate of molecular response, with no deaths 2 years after discontinuation.Read moreChapterDesigning and registering the reviewJanuary 2014A. BoothD. CraigNetwork meta-analysis should be done as part of a systematic review. Good quality systematic reviews, irrespective of whether they include pair-wise meta-analyses or network meta-analyses, are built on good design and careful planning. To reduce potential for bias, methods should be pre-specified in a protocol with subsequent deviations and changes from what was planned, recorded and explained ... [Show full abstract] adequately in the completed review report. In this chapter we describe the key features of a systematic review protocol and highlight areas which may warrant additional thought when planning a network meta-analysis. Transparent conduct and reporting enables those using systematic review and network meta-analysis findings to judge the quality of a review and assess for themselves the potential impact of any deviation from what was planned initially. We make the case why protocols should be available in the public domain and outline the role of systematic review protocol registration. We introduce PROSPERO, an open register designed specifically for prospective registration of systematic reviews. We look at how reviews with network meta-analyses have been registered and provide a step-by-step guide to the PROSPERO registration process.Read moreArticleSystematic review and meta-analysisNovember 2017 · Medicina Intensiva Miguel Delgado-RodriguezM. Sillero-ArenasIn this review the usual methods applied in systematic reviews and meta-analyses are outlined. The ideal hypothesis for a systematic review should be generated by information not used later in meta-analyses. The selection of studies involves searching in web repertories, and more than one should be consulted. A manual search in the references of articles, editorials, reviews, etc. is mandatory. ... [Show full abstract] The selection of studies should be made by two investigators on an independent basis. Data collection on quality of the selected reports is needed, applying validated scales and including specific questions on the main biases which could have a negative impact upon the research question. Such collection also should be carried out by two researchers on an independent basis. The most common procedures for combining studies with binary outcomes are described (inverse of variance, Mantel-Haenszel, and Peto), illustrating how they can be done using Stata commands. Assessment of heterogeneity and publication bias is also illustrated with the same program.Read moreLast Updated: 20 Jun 2021Discover the world s researchJoin ResearchGate to find the people and research you need to help your work.Join for free ResearchGate iOS AppGet it from the App Store now.InstallKeep up with your stats and moreAccess scientific knowledge from anywhere orDiscover by subject areaRecruit researchersJoin for freeLoginEmail Tip: Most researchers use their institutional email address as their ResearchGate loginPasswordForgot password? Keep me logged inLog inorContinue with GoogleWelcome back! Please log in.Email · HintTip: Most researchers use their institutional email address as their ResearchGate loginPasswordForgot password? Keep me logged inLog inorContinue with GoogleNo account? Sign upCompanyAbout usNewsCareersSupportHelp CenterBusiness solutionsAdvertisingRecruiting© 2008-2021 ResearchGate GmbH. All rights reserved.TermsPrivacyCopyrightImprint