I think you are correct in using ANCOVA. I probably wouldn't describe this as a pre/post design which can suggest a within-subject design. The design is consistent with an ordinary comparison of 2 treatment groups. Of course, you would have to acknowledge the limitation that you are not controlling for any other factors that might be temporally coincident with the shift in sedation protocol.

From authorities I have read, a quasi-experimental design (like a non-equivalent control group design) lacks randomness in its sample, hence an inferential analysis (like test of significant difference) would be inappropriate. If random sample was used, the "closest" design would be a t-test of mean difference between independent samples.

Eddie, I understand your point but there remains many (non-randomized) observational studies in medicine that can offer insights and do merit the application of statistical methods - so long as the design limitations are understood. For this study, I would advocate that ANCOVA or, if there are additional patient variables, propensity matching.

I would read the following discussion of the topic of pseudoreplication. And then read the literature related to W.G. Cochran and also Rubin on design of observational studies.

Cochran's work lead to the use of observational data to establish a causative link between cigarette smoke and lung cancer in litigation between the federal government and tobacco companies. Seems like a no-brainer now, but solidifying the scientific credibility of observational data to establish cause and effect was necessary to establish injury and damages in litigation.

The punch line from these works it that it's more about the conceptual model, study design and identifying multiple testable hypotheses than it is about selecting a statistical method. Causation from observational data requires testing multiple independent hypotheses that all lead in the "same" direction.

http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoas/1223908042

Cochran, W. (1965). The planning of observational studies of human populations. J. Roy. Statist. Soc. A 128 234–265.

Cochran, W. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics 2 295–313.

Cochran, W. (1983). Planning and Analysis of Observational Studies. Wiley, New York.

Cochran, W. and Cox, G. (1950). Experimental Designs. Wiley, New York.

http://www.masterenbiodiversidad.org/docs/asig3/Hurlbert_1984_Pseudoreplication.pdf

Thanks everyone!

Eddie I would like to be able to do a simply test of the mean differences in hours of mechanical ventilation between groups; however (as explained) there are (significant) clinical dissimilarities between the groups (not planned and simply by pure chance I suspect) - the pre-sedation protocol group are 'in general' sicker (SORRY GOT THAT ROUND THE WRONG WAY IN MY ORIGINAL QUESTION). If I take the finding 'as is' (without taking these clinical difference into consideration) - sure the protocol will look like it has done a fantastic job at reducing MV hours - but this is likely an artifact of the clinical difference in the two groups.

Looking at the design of this study and appropriate approached it does seem like an ANCOVA is the most suitable test. I was thinking multiple linear regression might work – putting the dichotomous treatment groups in as a predictor (along with other predictors) and MV has the outcome – any thoughts on this approach ?

With one group tending to be sicker on several dimensions, I would suggest a propensity scores approach. There's a SAS users group paper entitled "Using prepensity scores to adjust for group differenences: Examples comparing alternative surgical methods" out there that might help (among a large literature of accessible papers on this that you can Google).

Mark

I agree with Mark on ps-matching.

My only concern is when I hear Boonie mention "significant clinical dissimilarities between the groups." I've recently begun to fret about non-overlapping distributions. You'd really want to make sure that important variables had good 'support' across the two groups. Otherwise, control variables (and thus, by extension, PS matching) can't work.

To illustrate a pathological example: group 1 is mostly post-thoracic-surgery patients, middle-aged, in for a coupla days after elective procedures, end up with fewer MV hours >72. On the other hand, group 2 is mostly medical patients, older, and in for an exacerbation of congestive heart failure, end up with much higher prolonged MV.

Even if you enter a post-op dummy variable, or a CHF dummy variable, your analysis can't control for these clinical differences independently of the sedation protocol change. The control variables have insufficient support across the populations. PS matching can't deal with this, since the new PS variable would have the same support issue.

In this pathological case, the effect of the sedation protocol change is fundamentally unidentified, because of the pathological nature of the between group differences. You 'need' more of the post-op patients in group 2, and more of the medical patients in group 1, so that the distribution of your control variables is more balanced across the groups.

Rubin raised this issue in his 1997 Ann Intern Med paper. He illustrates it with a neat thought experiment: “… comparing 5-year survival rates among 70-year-old smokers and 40-year-old nonsmokers gives essentially no information about the effect of smoking or nonsmoking for either 70-year-old or 40-year-old persons.”

While your groups are unlikely to be so heterogeneous, if you're in a small facility and comparing relatively small periods of time with tight accrual thresholds, you could get some pretty different groups just by chance as you pointed out.

Best

Marco