It is surprising to find a statistician that selects a model with little or no data to support it. Why do you suppose your choice is correct when I could do the same thing and get an entirely different model. I suggest that in most cases data support is more important than theory support. To para -phrase Dick Feynman it doesn't matter how pretty a model is if it doesn't agree with experiment it is wrong Now that is science. David Booth

'In theory, theory and practice agree. In practice they do not.'

I see you're in biostats, so take my answer with it in mind that I'm in finance/economics. The preferred approach is that you first state what the real underlying question is, not "are these data correlated?" but something more like "Does real-world-event-A cause real-world-event-B?". Then specify some plausible, real-world mechanism for how A could cause B. It may be somewhat preferable to have it be a hard, mathematically defined theory, but often, if the original question is interesting, you can get by with simply a description of the mechanism.

Could you please give the example of your data?

For your data, I might recommend you consider the work of James (1983; 1984) for his linking of within-group agreement (Rwg) with theoretical rationale (null-distributions).

Hope this helps,

Matt

It is surprising to find a statistician that selects a model with little or no data to support it. Why do you suppose your choice is correct when I could do the same thing and get an entirely different model. I suggest that in most cases data support is more important than theory support. To para -phrase Dick Feynman it doesn't matter how pretty a model is if it doesn't agree with experiment it is wrong Now that is science. David Booth

David, you are right!

I would right out in hand writing three times over every little last assumption about:

-the experiment design

-compare your models assumption to other likely applicable models assumptions

-what was the criticism of your model?

-is the criticism of your model valid or plausible? [Note: I’ve have dehumanizing ‘professors’ trash talk my logic numerous times with no information provided to support their statements where they imply (without evidence or supporting argument) that my reasoning is flawed, they just wanted to be degrading towards me (and others) for unknown reasons – if this becomes a frequent thing it is a strong sign that you should switch departments and associate with better people]

-does there exist another well-developed more appropriate model?

-the randomization

-the grouping

-the pairing

-is there missing data

-the sample population

-the level of measurement

-the sample size

-the power

-the statistics

-the stability of that statistics

-the sampling distribution

-the phenomenon under study

-the known biological mechanisms which could affect your statistic

-the hypothesized biological mechanisms which could affect your statics

-brainstorm any confounding variables

-does the data appear to suggest your model is appropriate?

Matthew Robinson Your list is very comprehensive and I agree with it. From my understanding of the original posting the major problem seems to be in your last point and I agree again. What I don't understand is that the criticism is that the model lacks theoretical support. My reply to this comes from your last point. If the data suggest the model I don't care very much if there is a lot of theory support or not. In science Data Rules. Best, David

I'm not sure that data can be trusted alone.  If the model makes no sense from a subject matter perspective, have you made a new discovery, as I saw someone once note, or is it more likely a fluke which is not repeatable?  If you have a small sample, you are more likely to have an odd, "unrepresentative" result.  Further, I known that people might accidentally supply you with the wrong data.  If you cannot explain your model from a subject matter perspective, but you are certain you have adequate data, both in terms of quantity and quality, how useful is your model if you don't know what it means? 

I saw an example once, I think it may have been about the difference between correlation and causation, involving a model which suggested hydration drinks cause knee injuries, if I recall correctly.  But really, people who drink those drinks are more likely to play sports that sometimes result in knee injuries. But the drinks did not cause the injuries.  Well, suppose you have a model which theoretically needs frequency of knee injuries as an input, but you input volume of sports drinks instead.  That might work fairly well, but it may make little sense. A more complex set of such substitutions might occur.

So though data are very important, I think a model should be explainable. 

A case to consider might be Big Data.  You can't just take a huge volume of data and expect to explain anything consistently if you don't know more about the data source(s). 

Anyway, when a model is involved, I think it's best to be careful about both the data and the model. 

In the situation described for this question, I don't really understand the issue. But if a model is used and there isn't a reason which explains what it means, that does sound like something that should be explored. But, I'm just commenting speculatively.

Cheers. 

James R Knaub I believe there is nothing wrong with theory. But Data Rules. In any confrontation data wins. It must because it represents nature and that is what science is about. Your point about experimental blunders is well taken. That's why there is a current thrust to replicate experiments. But again in any contest, reliable data wins. Best wishes, David

I would not consider it a contest. If indeed the data are reliable, there should be a reason.