The ROBINS-I tool allows you to assess the risk of bias for a particular result (in fact as part of setting up the assessment you are asked to specify the numerical result being assessed).

So to answer your question yes, you can apply ROBINS-I to studies without mortality as their primary outcome.

The tool then asks you to consider confounding. Unadjusted results are likely to be assessed as being at higher risk of bias due to confounding than results which are appropriately adjusted.

It is a long document but I recommend reading the guidance for the tool which you can find here: https://sites.google.com/site/riskofbiastool/

Thank you for your answer. I would like to clarify my question with an example.

Consider a cohort designed to evaluate the effect of treatment with, let's say, cortisol to patients with pneumonia. The primary outcome of the study is mortality. The authors provide all necessary data (design, number of participants, confounding factors, outcomes etc) for the cortisol treated vs non-cortisol treated patients, and therefore a formal assessment of the risk of bias can be done.  

However, the authors also provide data (adjusted or not) for the subset of patients receiving specific antibiotic treatment (e.g. β-lactam+macrolide, fluoroquinolones, β-lactam+fluoroquinolone, β-lactam alone), which is the primary question of my meta-analysis. However, as expected, no other data are provided regarding this subset of patients.

Can I apply all the data that refer to the cortisol vs non-cortisol comparison to the subset of the e.g. β-lactam+macrolide vs fluoroquinolone treated patients, whose data have been included in my meta-analysis? From my point of view, the primary signalling question for confounding regarding the effect of interest cannot be answered with a yes or no, as there is no option for no information or not applicable for this bias domain. Therefore, assessment of confounding is not feasible. Similar issues may apply for the remaining bias domains.

I have posted for almost a week the same question to the Cochrane collaboration and ROBINS-I developing team. I had no feedback till now.

Hi Konstantinos,

I'm not very familiar with how CAP is treated so I'm sorry if I've misunderstood, but I think you are saying that some of the studies you are looking to assess are designed to compare mortality in cortisol treated patients vs. non-cortisol treatment patients, but some patients will have received specific combinations of antibiotics in addition to cortisol/no cortisol and you can extract the data for these participants?

Does this mean that you are calculating the effect measure between the antibiotic regimens yourself?

If the papers are as I imagine, I presume they adjust for confounders for the cortisol vs. non-cortisol comparison and are unlikely to have even made comparisons of the population characteristics of people receiving different combinations of antibiotics. 

I would say for the confounding domain (although this is very dangerous without seeing the paper) I would guess this makes risk of bias for the confounding domain critical.

If I've understood correctly, and you are having to calculate the effect measure yourself, that does raise an interesting question because I'm not sure if the tool is designed for that purpose, as I said previously you are supposed to specify the numerical result you are assessing the bias in- but in this case you are actually calculating the result. I'm not sure how you would be able to answer the other domains (again difficult without actually seeing the paper). But in any case if found to be at critical risk of bias on one domain the whole result is assessed as being at critical risk of bias.

Does this make sense/help?  

This is exactly what I think for confounding. However, the analysis is done per ITT, which according to the tool should immediately categorize the study to low risk for bias. And this is confusing.

So the question is can I use the tool for the assessment? Probably not, but I will wait for the Cochrane's view.

In any case, thank you very much.

Hi Konstantinos,

I'll be interested to hear what they say.

But, just for my information, how can a cohort study be analysed as ITT? 

I think that even with the Cochrane risk of bias tool for RCTs, a study can be found at high risk of bias on the randomisation or allocation concealment domains even if it is at low risk of bias on other domains because it performs an ITT analysis.

Final note: I would be wary of using any method of assessing quality/risk of bias without considering the result you are extracting. For the example you give, the study may be at moderate risk of bias (the best you can get for a cohort study [except in exceptional circumstances]) for the cortisol vs. non-cortisol comparison but at critical risk of bias when extracting other comparisons.

Good luck with your review!

I fully agree with you regarding the risk of bias in these non-randomized studies.

According to the ROBINS-I tool (and the ACROBAT-NRSI, their differences seem negligible to me), the first step when you start to assess the risk of bias of a non-randomized study is to "design" an ideal RCT that evaluates the same question as that of the study (and the review you are undertaking). So, when you are assessing confounding factors you must assess if the final analysis is done according to assignment in treatment (for an RCT this is the ITT) or to assignment and adhering to treatment (for the RCT this is the per-protocol).

It follows that due to randomization the majority of known and unknown confounding factors will be distributed equally between the compared groups in an RCT and even the cases that will quit, withdraw, die soon or receive part of the protocol treatment will be also similar. Therefore, the ITT analysis in RCT is considered without bias in relation to confounding factors. Based on this assumption, the same would occur in a non-randomized study. However, the lack of randomization results in the de facto presence of confounding factors. From this point forward, I cannot  explain how or why they consider the opposite for the non-randomized studies.

Thank you again for your time

Hi Konstantinos,

Applying ITT analysis to a RCT does not mean that there is no bias due to confounding. For there to be low risk of bias due to confounding, the randomisation has to be performed appropriately and allocation has to be concealed, in addition to people being analysed according to their treatment group (rather than according to the treatment they received).

With ROBINS-I, you are completely correct that you have to specify the target trial, which is the trial which would take the place of the non-randomised study if you were able to perform it. This is what you are going to assess the non-randomised study against. And as you say you have to specify whether you are interested in the effect of being assigned to a treatment (i.e. being offered something, which is what happens in a RCT) or assigned and adhering to treatment (i.e. actually taking the medication as directed). But if you are interested in being assigned to a treatment (aka ITT) it doesn't mean you can ignore the confounding domain when assessing a non-randomised study, because it is not a perfectly randomised RCT. In a non-randomised study, people will be offered/prescribed treatment generally on the basis of a whole load of prognostic factors which act as confounders. 

Being interested in the effect of being assigned to a treatment does mean that you don't have to worry so much about some questions, for e.g. in the domain about bias due to deviations from the intended intervention.

Hi Konstantinos,

just want to know , did you get any response from cochrane group. I am also working on a review that includes RCTs , quasi RCT and Non-RCT. I have seen many recent cochrane protocol/reviews (interventional) which included RCTs , quasi RCT and Non-RCT. No one has mentioned use of ROBINS-1. 

Unfortunately, I received no answer to my querry.