'In theory, theory and practice agree. In practice they do not.'

I see you're in biostats, so take my answer with it in mind that I'm in finance/economics. The preferred approach is that you first state what the real underlying question is, not "are these data correlated?" but something more like "Does real-world-event-A cause real-world-event-B?". Then specify some plausible, real-world mechanism for how A could cause B. It may be somewhat preferable to have it be a hard, mathematically defined theory, but often, if the original question is interesting, you can get by with simply a description of the mechanism.

Could you please give the example of your data?

For your data, I might recommend you consider the work of James (1983; 1984) for his linking of within-group agreement (Rwg) with theoretical rationale (null-distributions).

Hope this helps,

Matt

It is surprising to find a statistician that selects a model with little or no data to support it. Why do you suppose your choice is correct when I could do the same thing and get an entirely different model. I suggest that in most cases data support is more important than theory support. To para -phrase Dick Feynman it doesn't matter how pretty a model is if it doesn't agree with experiment it is wrong Now that is science. David Booth

David, you are right!

I would right out in hand writing three times over every little last assumption about:

-the experiment design

-compare your models assumption to other likely applicable models assumptions

-what was the criticism of your model?

-is the criticism of your model valid or plausible? [Note: I’ve have dehumanizing ‘professors’ trash talk my logic numerous times with no information provided to support their statements where they imply (without evidence or supporting argument) that my reasoning is flawed, they just wanted to be degrading towards me (and others) for unknown reasons – if this becomes a frequent thing it is a strong sign that you should switch departments and associate with better people]

-does there exist another well-developed more appropriate model?

-the randomization

-the grouping

-the pairing

-is there missing data

-the sample population

-the level of measurement

-the sample size

-the power

-the statistics

-the stability of that statistics

-the sampling distribution

-the phenomenon under study

-the known biological mechanisms which could affect your statistic

-the hypothesized biological mechanisms which could affect your statics

-brainstorm any confounding variables

-does the data appear to suggest your model is appropriate?

Matthew Robinson Your list is very comprehensive and I agree with it. From my understanding of the original posting the major problem seems to be in your last point and I agree again. What I don't understand is that the criticism is that the model lacks theoretical support. My reply to this comes from your last point. If the data suggest the model I don't care very much if there is a lot of theory support or not. In science Data Rules. Best, David