Hi Dirk,

I assume that when you say hierarchical modeling, you are including an empirical-Bayes type shrinkage adjustment, with the result that with low N, the estimate is shrunk towards the grand mean. I think this is a satisfactory outcome, but you report that hierarchical modeling does not work. Could you expand on what you mean by that; is it that with few hospitals and few cases per hospital you don't detect hospital differences? But that's to be expected. I don't think that your option 1 will be beneficial and while option 2 might improve the ability to detect the hospital random effect it will reduce the ability detect overtime shifts. Perhaps its the case that there just isn't sufficient information to reach reliable conclusions and that everything is working just as it should.

Mark

Just commenting so I can read other's thoughts on this.

Hi Mark and Amir

Many thanks for the quick responses and for your suggestions. My problem with applying HLM-Bayesian methods to this data set is indeed that the estimates are shrunken to the mean which makes sense mathematically but is not really satisfactory in terms of reporting outcomes back to the participating hospitals. For the same reason, option 2 is not a real option because we are expected to report annual results. Maybe there is no solution to this very special problem.

The number of units is about 30 with 75 percent of units with less than 30 patients (it's a point prevalence design, which is problematic, but this is how it is...) I will take a closer look at the paper you've recommended, Amir, thanks again.

With small sample sizes I think we should accept that we cannot say much on between center differences. Anything that we observe on between center differences might well be attributed to chance. This is the strength of "HLM-Bayesian methods" as referred to above, since we will see that all between center differences are shrunk to zero.

We have previously proposed that provider profiling should first adjust for chance by "HLM-Bayesian methods" (which we referred to as "random effect models"), before any further interpretation is attempted (PMID: 21642443). A second issue is whether we can sufficiently adjust for case-mix differences (which may rarely be convincing).

I do like the idea to pool the data over a some years to increase sample size, but as Mark Cohen observes, this reduces the ability detect over time shifts. Equivalently, we may say that we assume that outcomes are constant over a number of years. That is the price we pay in statistical analyses: we may learn something if we make stronger assumptions. If someone expects reports on annual results, I think methodologists have the task to explain that this will not be meaningful for between center differences.

Have you thought about a multilevel model? You should balance groups inside the cluster to fit the sample size requirements and you could probably think to use the case mix issue to obtain heterogeneity and not selecting for a specific health problem, since they come probably from the same ward.As long as I assume from your first comments.To pool data will let you lose the chance to investigate the difference in operators differences and organisational influence, which between different hospitals could be highly inconstant.

A further problem is that in pediatric populations adverse events can be much rarer than they are in adults. Have you tried modeling a composite outcome event that includes everthing? But its most likely that no model or approach you consider will be able to detect differences.

This is a very familiar debate! Some colleagues and I had similar discussions last year when preparing outcome reports for individual surgeons in the UK NHS. Some of our concerns are published in a recent Lancet paper.

The Lancet 2013; 382: 1674 - 1677

doi:10.1016/S0140-6736(13)61491-9

I would caution against any comparisons when volumes are small, even if mathematically plausible models exist.

After reading your comments and your papers (thanks to Ewout and Oliver), things get a bit clearer to me. The dilemma is to meet the external expectations and to deal with the methodological constraints at the same time. But this problem has to be solved in another field...

That's the "real" problem... There's this expectation that if data (any kind of data) are collected, the statisticians will be able to mine it for valuable information. Sometimes its just necessary to make the inferential problems explicit.

Hi Dirk

It is a difficult problem in general. What do we do with small sample sizes. A few suggestions. Is there an obvious split for types of hospital? Perhaps public or private? Then you could code hospitals into general types . Another way might be to use a weighting system which is applied to the patients you have in your sample and adjusts them up to the general (responding and non-responding ) patient populations by hospital. I would abandon hierarchical methods in your case for sample reasons.

Hi everybody

Thanks to our discussion, I will not conduct hierarchical methods but try to utilize O/E funnel plots with some risk adjustment.

@Marc: splitting of hospitals is not a real option, we are requested to report on every single hospital. But I definitely will consider the application of some weighting, thanks.

Hi, Dr. Richter

Although it is not clear for me whether you want to assess outcomes of pediatric units' or health patients' outcomes, I see this problem as not possible to solve using only quantitative approach. A qualitative approach (considering, for instance, story- telling from the point of view of managers and employees of such unities), combined with quantitative one (considering longitudinal series of patients with similar diseases, in past, compared with current situation of sampled patients), could give you a wide picture of the context before you go to your focused/aimed comparison.

Dr. Dirk. May you employ just a variable like incidents/patients for each year and use the average of the five previous years as a comparison referent? It does not explain too much considering particular conditions of each medical unit, but at least you might create a common annual report protocol as an initial step. Later each unit may decide if they weight the different possible incidents and how to do it in each reporting unit.