Joseph, thank you for the clarification.

Did I oversee your sentence in your first post: "If you see an effect then look at cost, hospital stay, etc."? I agree now completely with your very last statement that "Every effort should be made to disqualify the data." (after having seen a clear observed difference). I just think that's a mission impossible. It's that "etc." in your sentecnce I cited first. How can one know what this "etc." is? The problem here is that it is likely to be a considerable if not big issue.

But we (or I?) probably got trapped in a very academic distinction. Practically, it can be worth using the data (from a non-randomized study) to generate a hypothesis (carefully including all known facts about "cost, hostital stay, etc.") - I think here we fully agree - but by no means is it possible to test a hypothesis (about the treatment effect) and to conclude from the observed data that one treatment may be better than the other with a reasonable (or even quantifyable) confidence. Not sure if you agree with the latter, but I think so.

Proportion has nothing to do with it. Sample size does. see:

https://www.amazon.com/Elementary-Survey-Sampling-Richard-Scheaffer/dp/0840053614/ref=sr_1_1?keywords=Elementary+survey+sampling&qid=1559254345&s=books&sr=1-1

be sure to look at the sections on cluster and stratified sampling.

Best wishes, David Booth

i) I don't think it's a good idea to ignore data if you have it.

ii) what concerns me more is: are the patients really randomized to the possible treatments? If not, inference is meaningless.

iii) one more point: you seem to expect different variances under H0. Why? (not really relevant for sample sizes as yours, however)

Jochen is correct. Esp when you recall that R.A. Fisher said there is no experimentation without randomization.

This appears to be a retrospective observational study, neither cohort nor case control. Random selection is not possible, but inference will have a meaning if the effect is large and quantifiable.

There is no problem in the sample size gap if you use variance. Any comparisons with standard error will be biased. Variance (standard deviation) is what is needed to quantify the effect.

If you cannot see an effect without using statistical tests and manipulation, you do not need statistical tests and manipulation.

If you see an effect then look at cost, hospital stay, etc.

There is experimentation without randomization. Random is used to compensate for a poorly controlled experiment. Random is expected to correct for ignorance. Random sampling is a poor excuse for not knowing what you are doing. Nevertheless, sometimes it is all you have.

Joseph L Alvarez, it needs a lot of profound assumptions to put a meaning into inferences from non-random samples. I agree that it is possible, though, but it's usually not wise to do so, because it's extremely difficult to be aware of possible sources of bias. And bias can be extreme to any extend. If one had a sensible judgement on the possible size of the bias, an interpretation of an effect clearly larger than the possible bias would be ok, but it may be next to impossible to get an agreement on what the possible size of bias might be. When it comes to treating patients (with A or B, and maybe with other treatments not considered for the present analysis!), patients have a particular history and doctors consider a lot of features and contexts that determine the treatment. Thus, the treatment response can to any extend (100%, say!) be the effect of those things that lead to the choice of the treatment (->bias), and to 0% an actual treatment effect. I think it's hard to quantify what is bias and what is effect, and I think it's quite unsound (without a very deep and thorough analysis of *all* the things that happen until a patient sees the doctor and is assiged a treatment - something that will cost considerably more time and work that the actual "study") to claim that whatever difference one sees between the two groups is more or less a treatment effect.

Jochen Wilhelm you are changing the subject and contradicting yourself.

The study in question has to be retrospective. The sampling is already completed. Jane Pads does not say it is completed, but I doubt it is a prospective study of 1M and 20,000. The only way to approach it is to look for the effect. If one is not there, no amount of statistical teasing is going to find it. There is a great opportunity for p-hacking, but not much opportunity for science.

You say

"When it comes to treating patients (with A or B, and maybe with other treatments not considered for the present analysis!), patients have a particular history and doctors consider a lot of features and contexts that determine the treatment."

That is not random. But yes, sometimes a study will assign a treatment randomly. Why, if all these considerations were made? It is because enough considerations cannot be made. So, we assign with ignorance of all uncontrolled and unknown considerations in the hope that random assignment will give a population response. Few medical studies are large enough to give an ignorance-corrected population response. Cost and ethics preclude large, well-controlled treatment studies. Statistical significance is hardly a sufficient response

Dear Joseph, I don't see where I contradict myself (maybe you can explain your point). I was concerned that the patients were not assigned at random to the different treatments, and that in such a case inference can be drastically misleading. It's actually irrelevant if this data is retrieved retrospecively or prospectively. The only relevant point is if the patients were randomly assigned to treatments or if there are or have been "good resaons" to give a particular treatment to a particular patient.

Now Jane has (or collects or generates) a bunch of data and she sees some difference. If the assignment was not radom, then what I say is: even if this difference would be rather large, there is no way (no logically justified reason) to claim that this difference is caused by the different treatments. It can well be caused by things that lead to symptoms of patient characteristics that in turn practically determined which treatment was given (I call it "selection bias"). It can also be that the treatment differnce has the opposite sign!

You wrote that "If one [effect or difference] is not there, no amount of statistical teasing is going to find it.". My point is the analysis/interpretation of the difference that will inevitably be there (in the data). That observed difference can be a mere selection bias, and no amount of statistical teasing can rule that out.

Did I get you wrong? Where is my mistake?

Jochen, are you talking about bias by the physician on who is treated? I took your statement, "When it comes to treating patients (with A or B, and maybe with other treatments not considered for the present analysis!), patients have a particular history and doctors consider a lot of features and contexts that determine the treatment," to mean treatment is not random. I found this to be a contradiction.

Yes. Selection bias must be avoided in treatment and analysis. A prospective, case-control study should be well-matched between cases and control. It will be difficult to ensure well-matched when the number of participants is small and the condition of the participants has large variance. Random sampling is unlikely to ensure well-matched for a small study.

Paragraph 2 Now Jane ... An observed difference must be large before any consideration is given. Mere statistical difference is not enough, especially when the data were not purpose generated for the study. If the difference is large enough for consideration, then the data should be evaluated in terms of bias, type of treatment, etc. Every effort should be made to disqualify the data. It should not be qualified because it is random. Random might aid in selection bias if it is truly blind. Random does not qualify the assumptions.

Paragraph 3 You wrote ... Yes. An observed difference can be selection bias and any number of confounding factors. Every effort should be made to disqualify the data.

Joseph, thank you for the clarification.

Did I oversee your sentence in your first post: "If you see an effect then look at cost, hospital stay, etc."? I agree now completely with your very last statement that "Every effort should be made to disqualify the data." (after having seen a clear observed difference). I just think that's a mission impossible. It's that "etc." in your sentecnce I cited first. How can one know what this "etc." is? The problem here is that it is likely to be a considerable if not big issue.

But we (or I?) probably got trapped in a very academic distinction. Practically, it can be worth using the data (from a non-randomized study) to generate a hypothesis (carefully including all known facts about "cost, hostital stay, etc.") - I think here we fully agree - but by no means is it possible to test a hypothesis (about the treatment effect) and to conclude from the observed data that one treatment may be better than the other with a reasonable (or even quantifyable) confidence. Not sure if you agree with the latter, but I think so.

Jochen, we agree except for data from a non-randomized study. There are many non-randomized studies that are gold-standard. Smoking and lung cancer is one. When the effect is obvious, it is a no-brainer. The distribution of reported effects ranging from no-brainer to statistically significant is highly skewed.

Observation of an effect (measurement with low uncertainty) is subject to disqualifying by ... The first question to ask is how large a confounding factor is necessary to produce the effect? The next question is what can it be?

As R A Fisher once said there is no experimentation without randomization. See R A Fisher, Design of Experiments, D C Montgomery, Design of Experiments, John Lawson, Design of Experiments, and the list goes on and on. Best regards, D. Booth

However I should have said also that there are some questions that researchers wish to study for which randomization is not possible. In this case we must give up standard inferential statistics because that subject depends on probability samples. The best we can do is there is to try somehow to approximately apply standard inferential statistics. Here you see people who suggest things like imagine a population from which a simple random sample like that which we have could have been drawn. These suggestions are mostly unsatisfactory. The only thing left then is to treat the collection of data as a population. Here the question is how do you generalize the calculations to the real population you are interested in. It seems to me that one is much better off using randomization from the start. If this is truly impossible and we cannot argue that approximation by standard statistics is possible we must result to much more undesirable methods which have as yet not been clearly explained in my opinion.

Finally the gold standard in medicine is a randomlzed clinical trial. While some might be desirous of moving away from that standard we do so at our peril because in the nonrandomized case we have no idea what the probability is that we are correct in our conclusions.

Joseph, as you mentioned smoking as example:

The effects of smoking on health was disputed for a long time. The observation generated the hypothesis that smoking might not be harmless, but only after a better understanding of the biochemical and physilogical processes was developed (from randomized cell culture and animal experiments), it was accepted that smoking in fact causes diseases. That was much more the result of accepting the analogy and the plausibility. By the way, it is very interesting to read RA Fishers responses to that topic (he was paid by the tobacco industry!).

Today we do have animal model of smoke-induced diseases (like COPD). These are randomized. Again, the analogy that the processes in humans are similar is still required, but adding what we know from biochemstry and cell physiology from mouse and human cells, noone would reject to believe that smoking is harmful. All strong, reliable arguments came from randomized experiments.

In medicine and epidemiology, the famous essay by Sir Austin Bradford Hill is usually cited when people discuss association vs. causation.

• https://www.edwardtufte.com/tufte/hill

Here are a couple points that I think are noteworthy.

Finally, for those who are about to point it out, I am aware that there have been many critiques of Hill's essay. I am not arguing that it is the final word on the topic. But I think it remains a useful starting point. HTH.

Jochen, the first Hill study was case-control, opportunistic. The larger doctor's study was case-control, opportunistic. There was much controversy about the results with especial tobacco support by Fisher. Fisher did not object to lack of randomization. Fisher claimed confounding. Cornfield in 1959 showed there was no confounding factor that was large enough to account for the effect. The effect was a factor of 10 higher lung cancer for smokers compared to non-smokers. Laboratory studies were not necessary to establish the association. Indeed, there are those who still dispute the lung cancer and cancer association.

I do not dispute the need for randomization in both clinical and animal trials. Randomization is necessary to prevent bias, whatever the source. I do dispute that randomization can control for ignorance. Perhaps it can on a very large trial with a large effect. It cannot on a small trial with a small effect.

Joseph, I think the key of your "dispute point" is that you consider a particular study (an actual set of data), but the statistical properties are about the procedures. You wrote "Perhaps it can on a very large trial with a large effect. It cannot on a small trial with a small effect." (emphasis mine). You use "trial" in singular. I totally agree: randomization does not ensure the absence of bias in any particular case. I am not sure if one could say that a particular study is biased. Bias is a systematic distortion, and there need to be something systematic. I can only have a rule that will cause bias (like the rule to treat young patients preferrably with drug A but old patients preferrably with drug B). This rule will produce studies where the drug effect is confounded with age and so possibly bias the results for the drug effect (so I don't know what the difference is when Fisher says he is concerend about confounding, but not about bias - one is the reason for the other, as far as I understand).

If I follow my bad assignment rule, I can expect systematically wrong estimates (if interpreted as being the effect of the drug and if age has an impact on the outcome). It's not about a particular study, it's about expectation. Randomaziation ensures that I can expect the correct estimate, even if age and who knows what else had an effect).

PS: Thank you for that lifely discussion!

Jochen, definitions do sometimes confound us. Confounding is an unconsidered factor. A series of trials can be systematically confounded; therefore biased by confounding. I use bias to mean intentional or unintentional effects caused by the researcher. One trial can be biased by the researcher.

My dispute is with randomization to control ignorance. Fisher used randomization to control for ignorance. An agricultural trial requires a large area for testing. A large area can vary greatly, despite an appearance of uniformity. Soil type, depth, drainage, ... will produce a non-uniform crop. Small plots are made within the trial area. The small plots are randomly assigned treatment. The researcher is ignorant of the much of the soil and irrigation details, but can assume uniformity. Random is the control for non-uniformity. Random is the control for ignorance.

The complication of many investigations requires that we choose ignorance to simplify the trial and control ignorance by random sampling. Perhaps it can control for ignorance on a very large trial with a large effect. It cannot on a small trial with a small effect.

Indeed, a very good discussion.

I would probably not say "to control for ignorance" but "to acknowledge ignorance" - by literally placing the ignorance into the experiment (to ignore anything but the experimental factor when selecting plots or individuals).

In PDQ Epidemiology, Streiner & Norman point out that authors sometimes fail to make a clear distinction between random sampling and random allocation. But as they say:

"Bear in mind that random selection and random allocation have different aims. As stated earlier, random selection of subjects is used to ensure the generalizability of the results from the sample to the population. The purpose of random allocation is to ensure the validity of any cause-effect interpretations we make from the data."

Source (I hope this long link works):

https://books.google.ca/books?id=PQ54T8WHyl4C&pg=PA53&lpg=PA53&dq=%22Bear+in+mind+that+random+selection+and+random+allocation%22&source=bl&ots=Tyuf6S4mfj&sig=ACfU3U31lDlAyMa2SR_Ma5jU6vnU1MVhNQ&hl=en&sa=X&ved=2ahUKEwiejJin-83iAhVoTd8KHbgdBekQ6AEwAHoECAAQAQ#v=onepage&q=%22Bear%20in%20mind%20that%20random%20selection%20and%20random%20allocation%22&f=false

You acknowledge ignorance so it is necessary to compensate or account for ignorance. You compensate or control by random sampling, because there is no way to account for ignorance other than widen the uncertainty by estimation. It is control because the objective is to make a controlled prediction. A controlled prediction is a prediction within the uncertainty bounds.