Interesting concept Frederik. I think it is an ambitious project that can potentially give interesting results. I noticed that your search criteria "searching for "randomized controlled trial" will give you around 400000 hits, and looking quickly it is easy to see that there is a lot of non-drug trials. For newer drugs the assessment reports from the authorities (EPARs from EMA here in Europe) will tell you about what pivotal trials were assessed when marketing autorisation (MA) was given. Also clincial trials databases might be a better place to search for truly relevant trials. 

Also I wonder what you should do with trials with more than two arms , so with treatment, comparator and placebo which I suspect you will see being quite common. Should these be excluded or included? 

Finally I would suggest that you consider if random sampling is the best way or if selecting drugs/drug classes could be better. I am not sure but I suspect that for a pilot type of study that could be better. Drugs where real-life efficacy is less impressive that their RCT effects for instance.

Dear Frederik,

In most clinical trials, it is the assessor which needs to be blinded. With an adequate clinical trial design, side-effects and adverse effects experienced by the participants (patients or volunteers)  are noted and  monitored ideally by nurses/medics -unrelated to the primary Trial assessment outcome- and kept secret from the principal assessor for the duration and closure of the study.

Moreover, this assessor bias becomes important only in trials with subjective measurement scale outcomes.

Many clinical trials of candidate drugs against unmet diseases rely, require and depend on objective measurement scale outcomes.

The reference you provide focuses exactly on these points (assessor bias and subjective measurement):  

"We provide empirical evidence for observer bias in randomized clinical trials with subjective measurement scale outcomes. A failure to blind assessors of outcomes in such trials results in a high risk of substantial bias." "However, these trials are not representative of medical trials in general."

I attach the relevant sections from your reference and a corroborating section from wikipedia on blinding in randomized control trials .

I cannot see the relevance between volunteers finding out by experiencing medication side-effects if they are on placebo or not  with the validation or randomization issues in clinical trials, or even the blinding procedure where it counts (i.e the assessing doctor).

-----------------------------------

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3589328/pdf/185e201.pdf

Interpretation: We provide empirical evidence for observer bias in randomized clinical trials with subjective measurement scale outcomes. A failure to blind assessors of outcomes in such trials results in a high risk of substantial bias.

Limitations.  The trials we included in our analysis are contemporary and represent a variety of clinical specialties, and their design implies a low risk of confounding. However, these trials are not representative of medical trials in general. We included no trials with clearly objective measurement scale outcomes, such as non repeatable automatized laboratory measures. The included trials had subjective outcomes, and our results apply only to similar trials.

Conclusion. We provide empirical evidence for observer bias in randomized clinical trials with subjective measurement scale outcomes. Failure to blind outcome assessors in such trials results in a high risk of substantial bias.

--------------------------------------------- 

 http://en.wikipedia.org/wiki/Randomized_controlled_trial

http://en.wikipedia.org/wiki/Randomized_controlled_trial#Blinding

Blinding

An RCT may be blinded, (also called "masked") by "procedures that prevent study participants, caregivers, or outcome assessors from knowing which intervention was received."[46] Unlike allocation concealment, blinding is sometimes inappropriate or impossible to perform in an RCT; for example, if an RCT involves a treatment in which active participation of the patient is necessary (e.g., physical therapy), participants cannot be blinded to the intervention.

Traditionally, blinded RCTs have been classified as "single-blind," "double-blind," or "triple-blind"; however, in 2001 and 2006 two studies showed that these terms have different meanings for different people.[48][49] The 2010CONSORT Statement specifies that authors and editors should not use the terms "single-blind," "double-blind," and "triple-blind"; instead, reports of blinded RCT should discuss "If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how."[3]

RCTs without blinding are referred to as "unblinded",[50] "open",[51] or (if the intervention is a medication) "open-label".[52] In 2008 a study concluded that the results of unblinded RCTs tended to be biased toward beneficial effects only if the RCTs' outcomes were subjective as opposed to objective;[46] for example, in an RCT of treatments for multiple sclerosis, unblinded neurologists (but not the blinded neurologists) felt that the treatments were beneficial.[53] In pragmatic RCTs, although the participants and providers are often unblinded, it is "still desirable and often possible to blind the assessor or obtain an objective source of data for evaluation of outcomes."[34]

Ref46:  http://www.bmj.com/content/336/7644/601.full

For drug therapy using placebo there is however an issue that the blinding provided for the patient can be compromised, either by some immediate effect or potentially by side effects experienced recognisable as belonging to the drug. This can also affect the nurse/physician involved with the patient. To me it would seem that the outcome will decide if this has a potential for differential reporting of effect from the patient, if the drug has subjective effects to be measured (so assessors role in deciding effect is small) the patients belief about what treatment is received can have an effect that the study design would have trouble dealing with. The magnitude of this issue is a bit unclear but given that there are several different criteria to be met for it to be an issue, it is probably limited to specific type of drug-disease combinations. 

To complicate (or simplify ?) the matter even more: there is also a possible nocebo-effect. I vaguely recall a study some 30 years ago, an on anti-depressant. Several patients in the placebo group experienced AE that were inquired on or mentioned as possible. Nocebo-effects are limited to functional symptoms in the realm of psycho-somatics, e.g. nausea, vertigo, itching, fainting, blurred vision... Perhaps it is advisable to differentiate your protocol between objective and subjective AE ?

Hi Kristian,

I like the idea to start from EMA's marketing's authorizations. It would increase relevance of findings. I just checked a few recent reports.However, I find it time-consuming to skim trough the text and catch pivotal placebo-controlled trials. Also references to publications seem missing. Therefore I worry that this approach may be too ambitious. Do I see this correct? Or how would you/others make this feasable? 

As for clinical trial databases, you mean CENTRAL? Here you would find non-pharmacological results too. No time saver compared to PubMed? ...There's also https://clinicaltrials.gov ... anything possible here?

I think for marketing authorizations two arm RCTs are fairly common. As for more than 2-armed RCTs: I have no scientific rationale -in terms of potential effects of AEs on unblinding -so far to include/exclude these studies. Anyone else does? 

Drug classes may be a good focus. We could choose for the most commonly prescribed drug classes. This would make things more relevant and maybe ease out the search and inclusion process as compared to EMA's site documents' skipping. Or as you suggest, real life drugs that perform worse than in RCT environments. Where to find such overviews?

Hi Yannis

Thanks for your thoughts! If I understand you correctly, this is your main issue:

I cannot see the relevance between volunteers finding out by experiencing medication side-effects if they are on placebo or not  with the validation or randomization issues in clinical trials, or even the blinding procedure where it counts (i.e the assessing doctor).

The relevance for me lies in the possibility that patients who find out there on placebo may become discouraged and report lower effects to an otherwise perfectly blinded outcome assessor. Vice versa, finding out they are on the drug may foster enthusiasm (enhance expectations).

I think randomization -if properly done- is a valid intervention. No doubts. It makes sure we give treatments randomly and not by e.g. patient preference. 

I agree we must think about what outcome types to include. And make as broad as possible inclusion strategies. Clearly subjective outcomes are part of this. I don't think a more objective outcome, such as blood lab test will be influenced by unblinding. But - this is my critique - I want health care to focus on clinically-patient relevant outcomes. So ideally we would focus on these outcomes. But  it is a challenge to define relevant and non-relevant. 

Hi Willem

My opinion here, but maybe off-topic as I would like to proceed with the protocol drafting :)

Subjectively created AEs by a nocebo effect are the price we pay for informed consent. Not mentioning all possible AEs during study enrollment would raise opportunities to know the real AEs from a drug. I suspect many drugs nowadays are safer than perceived. My wish would be informed consent without mentioning AEs. And avoid conditional learning of AEs. More ethical. 

"My wish would be informed consent without mentioning AEs." If the researchers did not mention potential side effects it would not be considered informed consent. Additionally, by the time a drug is in phase 3 or 4 trials the researchers will already have some idea of the most common/likely side effects from the initial safety and dose ranging studies - withholding this information would be unethical (re: violating the informed consent requirement) and possibly dangerous (where early recognition of an adverse event can prevent serious sequelae. 

Hi Frederik,

Re: "But - this is my critique - I want health care to focus on clinically-patient relevant outcomes. So ideally we would focus on these outcomes. But it is a challenge to define relevant and non-relevant."

I agree with you that outcomes must be patient relevant - there is no point of an intervention if it doesn't in some way benefit the patient, (or perhaps their carers).  However, there are difficulties with more subjective outcomes such as HRQoL or tools to capture symptom severity etc - with patient unblinding one of a great number of hazards.  If we accept that there are often objective outcomes that are unlikely to be altered by detection bias or performance bias (which may be affected by unblinding), these probably should be our focus.  And there is no reason why these outcomes cannot be tested to see whether they are surrogates for patient relevant outcomes.  A framework that can link an objective outcome (such as a lab test or a combination of tests) to the likely changes to quality of life experienced by a patient might be a better approach than attempting to solve the impact of unblinding upon the reporting of subjective outcomes.

Just a thought.

Hi Kevin, I used to see ti your way. I see this different now. Let me explain.

In Phase 3 or 4 trials, we know a drug is fairly safe (we tested for safety in phase I and II). We want to confirm large scale effectivity and catch some less common AEs. One could argue that in order to keep confirmatory trials adequately blinded, it is better not to mention common/likely side effects. This way we would capture true AEs frequencies (and not nocebo induced ones. cfr. Willem's supra) AND we would have a more precise estimate of effectivity. Knowing the true effectivity and harm of a drug would be a good starting point for delivering marketing authorisation and widespread use in large populations. This is what I call ethical science.

You still are not able to obtain INFORMED consent without listing the likely side effects. Unless you want to change the definition of informed, it is not possible, and thus, unethical. We have a long history that demonstrates the problems with allowing partially informed consent in medical research. While the vast majority of researchers may not misuse the wiggle room that partial information provides, the risk is great enough that the benefit is not worth the risk.

I understand your desire to use objective outcomes totally. However, I think we have seen some of the problems with using surrogate outcomes in such areas as cardiovascular medicine and diabetes. Our early obsession with cholesterol numbers, for instance, led to us vastly overestimating the benefit of statins. While the number needed to treat to lower cholesterol with statins is small, the number needed to obtain any real benefit (i.e. prevented heart attacks) is order of magnitudes greater. And for diabetes we have to only look at rosiglitazone for a drug that did a great job of lower blood sugar, while raising the risk of heart disease -  the actual outcome we are trying to prevent in treating type 2 diabetes.

I would also suggest that even phase 3 and 4 trials are, generally speaking, too small to detect most rare adverse effects. Long term pharmacovigilance is the only real way to detect rare outcomes. Of course, we may be talking about very different  levels or rarity. 

Overall, though, I do understand your frustration with the difficulty in knowing which adverse effects are real, and which are psychosomatic or attributable to non-drug effects in a patients life. This is entirely the reason for placebo control, or using an alternate treatment. There are also trials that use an active placebo (one that produces side effects), although finding a suitable active placebo is extremely difficult ( I imagine).

Yes, ethical science is that which provides real information on the benefits and risks of a treatment, but we cannot be more ethical by sacrificing informed consent. It is too important. We have decided, as a society, that the needs of the many are not justification for misusing the few. Your moral calculus may permit this, and I can understand it, but we would need to rewrite our ethics to make partial consent okay.