Bivariate models are probably a mistake because they assume implicit thresholds while the true reason is probably spectrum effects (see A comparison of univariate and bivariate models in meta-analysis of diagnostic accuracy studies. Foxlee N, Stone JC, Doi SA. Int J Evid Based Healthc. 2015 Mar;13(1):28-34.). Additionally, when you run the univariate Se and Sp its better to avoid random effects models as they are also a mistake and suffer from overdispersion and large MSE. Better to run your analyses using the IVhet model available in MetaXL (download from www.epigear.com and see Advances in the Meta-analysis of heterogeneous clinical trials I: The inverse variance heterogeneity model. Doi SA, Barendregt JJ, Khan S, Thalib L, Williams GM. Contemp Clin Trials. 2015 May 20. pii: S1551-7144(15)30007-0. doi: 10.1016/j.cct.2015.05.009.

The bivariate model includes a parameter to model the threshold effect, so I am not surprised by your comment that some of the heterogeneity appears to be accounted for in data with a threshold effect. However, you don't seem to have many studies for this kind of analysis. The bivariate model has 5 parameters, I do not think you will get good estimates, if any, with only 4 studies per meta-analysis. Are you using a Bayesian method? If so, are your results sensitive to the choice of prior? I don't see how you could get convergence if you were using SAS PROC NLMIXED or similar with only 4 studies.

Suhail, Im not sure I quite understand your answer. You are stating that the bivariate model is not appropriate for my situation? Im convinced there is no spectrum effect on my data, because Im working on a data from a field where there is plenty evidence that the disease spectrum does not interfere in the amount of antibodies in a subject. Therefore, spectrum is not a determinant of serological tests results. However, the risk of bias assessment showed a high fraction of either unclear or high risk in 6 of the 7 dimensions of QUADAS-2. Also, although not explicit, Im sure that the original serological tests in the meta-analysis did not use the same decision thresholds, as all the manufacturers recommend that this limits must be estimated in each run of the test considering values of internal controls. Therefore, even in a single test in the same research with several runs the decision threshold may change substantially. When you said I should avoid the random effect, did you mean the bivariate model or the D&L model? It seems interesting the reading you suggested. I will take a look, thank you. However, Im not up to (Im not good enough) programing new functions when they are not available in the software. 

Francesca, Im using the mada and meta packages of R-project. The reitsma function is described as linear mixed model with known variances of the random effects and variance components are estimated by restricted maximum likelihood (REML) as a default. But it mentions extensions of Reitsma's approach and Doebler transformations, and Im able to plot and SROC from it. Therefore, Im believe it might do stuff a little different the originally proposed by Reitsma. The functions does not ask priors in any of its arguments, therefore I believe this function does not allow an bayesian approach. Nevertheless, it is true that in the nine groups I had with 4 or more studies, two returned error of convergence. 

The function documentation is available in 

http://finzi.psych.upenn.edu/R/library/mada/doc/mada.pdf

http://finzi.psych.upenn.edu/R/library/mada/html/reitsma.html

But my doubt remains. A pooled sensitivity and specificity estimate by the bivariate model is interpretable even if there is a moderate/high evidence of heterogeneity? My hunch is that if the threshold effect is present and no other source of heterogeneity is detected, the model results interpretable. However, Im not sure if otherwise. Nevertheless, the review Im working on has so many other issues related risk of bias and absence of data other than Se an Sp, that one may challenge if meta-analysis may be appropriate at all. Of course, this will be discussed at some point.  

Pedro, what I am saying is that the theory that there are normally distributed random effects where each study has a different true effect is a myth. If you really have differences in true effects eg because thresholds are defined differently, then you need to subgroup studies such that thresholds are not too dissimilar before univariate meta-analysis. In this situation you will then have only two residual causes of heterogeneity: spectrum effects (which you do not have) and systematic error. For the latter you have two options: Run a IVhet or a QE univariate meta-analysis of Se and Sp separately by threshold subgroup. You do not need any technical expertise to do this - just use the excel plugin available from www.epigear.com

Why I say the bivariate model is not applicable in your case was that I assumed that you had spectrum effects instead of implicit thresholds. Even when you do not have spectrum effects, the list of studies you have contain researcher defined thresholds, so even if thresholds do vary, it would be unreasonable to assume normally distributed true effects for your studies. The bottom line with bivariate models also is the fact that comparisons of the summary Se and Sp from such models are more or less the same as univariate results through random effects models so the additional statistical complexity does not really generate any additional tangible benefit. Finally, if you read the paper I indicated previosly, there is good reason why random effects models in general should be avoided in meta-analysis.

I know this may be difficult to appreciate given the fact that I am querying the validity of recommendations that organizations such as Cochrane have  put in place. But even if you decide to run the conventional analyses, please keep in mind the controversy that I am alluding to.

Finally, if you would like to share data, I would be happy to run a preliminary analysis for you using our models - my email is on the university website

Hello Pedro, I believe the interpretation problem of sens and spec from a bivariate analysis relates to the fact that if the original studies are using different thresholds, then it's very hard to tie the estimates you get to a particular threshold, so you cannot make recommendations about which threshold people should use. I think it's really a problem of lack of clinical applicability. Re non-convergence, I think that's a sign you do not have enough data to fit the bivariate model, and so univariate analyses (whether random effects or not) may be your only option. I also think that convergence by itself is not a sufficient indicator that the analysis has gone well. A last thing, whatever method you use for meta-analysis, I think it's important to model the within-study variances as binomial (not all packages do this) as it's more accurate. Some packages use an approximation to the binomial within-study variance that only really applies with very large samples.

Also, I admit that I too do not quite follow Suhail's argument against random effects. Suhail - could you please expand on your statement, "theory that there are normally distributed random effects where each study has a different true effect is a myth. "?

Hi Francesca, yes sure, I can expand on that. Lets look at it for simplicity's sake by considering only the univariate pooling of say Sensitivity (Se). Each study reports an effect say p_hat and that is an estimate of an underlying true effect p (where p is devoid of random error). The random effects model assumes that if there are for example 5 studies and thus 5 true effects (i.e. p1 ... p5) , these 5 values of p follow a normal distribution with variance tau sq.  This is what is the myth. We do not sample studies in a meta-analysis randomly from a super-population of studies and this assumption goes against the very clear requirement of randomization in statistical inference.

Statisticians have argued that while it may be inaccurate, it is not an "unreasonable" assumption. My group has shown that it is indeed unreasonable because it:

(1) increases the variance of the estimator (yes, the random effects estimator has a greater variance and MSE than the fixed effect estimator regardless of heterogeneity).

(2) makes it impossible for the variance calculation to be correct and thus whatever variance we compute for the pooled random effects estimate, it is too small and thus we have overconfident results

In summary, all these problems arise from the myth that true study effects vary randomly and follow a normal distribution with a common variance. That is why I advised Pedro that if he is planning on executing univariate meta-analysis, he should avoid random effects models (given that there now exist plausible alternatives).

Thank you very much Suhail. I think I will need to read your paper to understand your arguments properly. It may be that the assumption of a Normal distribution for p1 ... p5 is not ideal, but I would also have a problem with the assumption of fixed effects of a common p for the 5 studies. Given the frequent methodological and clinical differences between diagnostic studies, I do not think the assumption of a common p can be the first choice. At this point I should declare myself to be one of those statisticians. As George Box wrote, "essentially, all models are wrong, but some are useful." I do have problems with some aspects of the bivariate/HSROC model, but the Normal distribution is not one.

The problem of the sample of studies not being representative of some theoretical population of studies is a problem for all kinds of meta-analysis, whether diagnostic or not, or using random effects or not. I am sure Pedro is aware of problems due to publication bias etc.

I do not quite follow your points (1) and (2) above. (1) seems to be saying there’s a problem with random effects because the variance estimate is bigger, and (2) that there’s a problem random effects because the variance estimate is too small. Is this not an argument in favour of random effects over fixed effects? Especially as in practice the point estimates from each kind of analysis may well be close, i.e. use the more conservative method.

Anyway, as I said before, I think I will need to read your paper. And I am sorry if I have prolonged the discussion beyond the point of usefulness. It’s just that I do not often get to have a conservation about this topic, which is of interest to me!

No, thats fine. I think the point of confusion is the "true variance" versus "model derived variance". Every estimator has a true variance (determined through simulation) but we also need to "predict" the variance through a statistical model (to generate a confidence interval around any generated point estimate).

For random effects estimators, the true variance is always larger than the true variance of the fixed effects estimator (regardless of heterogeneity) - this is bad, That was my point (1).

Of course the model derived  variance under the fixed effect model is too narrow and under the random effects model it is wider. But no matter how large the model derived variance gets, it will always be smaller than the true variance. Thus, the random effects model remains with the same problem the fixed effect model has under heterogeneity - this (as you are aware) is called overdispersion in statistics and was my point (2).

Hello Suhail, I am still unable to reach your conclusion (and still have a general preference for random effects, particularly for diagnostic studies), but there may now be a pause while I try to find the time to read your paper.

Are there exact definitions of "true variance" from simulation and "model derived variance" in your paper? After all, simulation also requires a model.

Yes, in simulation you generate a meta-analytic pooled effect say a thousand times and you compute the variance for each iteration ie each of the 1000 meta-analysis through whichever model and compute the coverage probability (ie you know what the true effect is and therefore what percent of times the confidence interval includes the true effect. This tells you if there is overdispersion or not. This is the model variance being assessed.

You compare estimators by directly computing the variance over 1000 repetitions directly ie in each iteration compute fixed and random estimates and generate the variance directly from the 1000 estimates for each model. This is the true variance being compared.

I should probably add one more comment. We are discussing the statistical failures of the random effects model. But lets look at it from a common sense point of view - all that happens in meta-analysis with random effects models is that larger studies are progressively penalized and smaller studies are rewarded (in terms of weight) as heterogeneity increases and this behavior ceases when we reach the arithmetic mean (equal weights). I doubt if anyone has justified this behavior of the random effects model presumably because it cannot be justified.

Well, this post was good. Respectful comments of different points of view and I had a chance to read and learn a few things as for long I did not. 

Concerning the original question, the answer I would say to myself now is that the bivariate model does not "account" for the heterogeneity, but it returns conservative Se and Sp estimates as it usually estimates larger variances an consequently wider confidence intervals. If threshold effect is present, then it may account for the heterogeneity of the threshold effect. Therefore, it seems to me that the bivariate model accounting for heterogeneity it is more like a conceptual workaround than a solution.  

Additional comments:

Im aware about the random process involved in the statistical deduction and inference. Nevertheless, I would say that in 90% of all investigations in the medicine field this principle is either ignored or avoided on purpose with some convincing arguments. In diagnostic test research, subjects are usually included either in case-control fashion or sequentially, in clinical trials, subjects are usually included consecutively and later randomly allocated to different interventions, in cohort studies inclusion usually occurs given an exposure or consecutively like trials. Nevertheless, we use statistical techniques to give data some interpretation. In the same way, it is clear that the studies included in a meta-analysis are a not random sample from all possible studies, but one may understand, as I did long a go, that it is a sample. This understanding may come easily from a funnel plot and publication bias discussion. One may use arguments to "flexibilize" assumptions of a statistical technique to give interpretation of a result. For example, last year a conducted a prediction model with a ordinary least squared linear model. I knew the outcome was not normally distributed, and other options were also problematic. At the end the predictions were at least fairly accurate and calibrated and, most of all, it traveled for other data. Most of my colleagues accepted the assumption breaking as main results were convincing. Im not sure if I got a place in hell for that. 

Another point that I am always concerned with and Francesca pointed it well, is the giving sophisticated statistical treatment to rubbish data, and  in the meta-analysis case, it usually means to combine stuff that should not be combined. There is a very thin line between trying to give results some interpretation, where risk of bias and heterogeneity is present and to pool summary estimate to please our peers in review process and ignore the risk of bias and heterogeneity. This is and important thing to always keep in mind. 

Thank you guys. :-) 

Thank you :-)

And thank you.