I am running a linear regression model and P-value after putting all the covariates together in the model gives significant value. Is there a possibility that an excess number of covariates in the model could produce artificially significant p-value? Is there a way to check if the p-value we got is truly significant and not artificial?
There is nothing like "artificially significant" or "truely significant".
Significance relates to given data under a given model.
Jochen Wilhelm how about the condition in which the percentage of missing values in around 10%. Would we change the model then? or if we don't change the model how would we describe the effect of missing values on results in our paper?
Hasan Bin Sohail
Your model is significant due to the covariants. I would suggest you to test the initial model without covariates and the problems of autoregression, heteroskedasticity, and the endogeneity of the model without covariates to check whether there is a significance and if there is no significance then your model may be invalid.
1. Add new main variables or increase sample data or
2. Confirm that the model is responsive when covariants are added to the model.
Note - There are no artificially significant" or "truely significant" terms in statistics to my knowledge.
if you require any assistance please do drop a messgae
Good Luck !!!
If the values are missing completely at random (MCAR), there is no problem. Otherwise your conclusions can be biased. The best option is to understand why which values are missing and to consider this in the interpretation. You might try imputation techniques to see how sensitively your results depend on the missing values.
Your choice of covariates should be based on the particular model that you are interested in the particular conditional hypothesis that you want to test. If you get differences in p values when some covariates are included, it is not that any p values are wrong (or "artificial"), it is that they are for different models/hypotheses. In many cases the "throw all the possible covariates in" approach means any individual coefficient is impossible to interpret (my brain can't understand the effect of x1 conditional on linear effects of x2, x3, x4, x5, x6, etc., particularly if these are correlated). Is their inclusion based on a particular theory? Of course now you have done that, and used up your p value.
Daniel Wright Thank you for your comment.
The inclusion of covariates was based on previous literature. I did not use the approach of "throw all possible covariates in".
I made four models and included covariates step by step. However, only in the last model after including certain covariates, I got the message in output that 200 observations have been deleted due to missingness. So this means that the problem was in those covariates included in the last model. These 200 missing values make around 10% of the total observations. So I decided to keep the results as such and explain that while writing a paper.
The issue is I didn't find a significant association of temperature with cardio admissions in three models but the association was significant in model D.
But now my question is how would I explain that in the paper? Would I write like this: The results in the last model (let's say Model D) might be biased because of 200 missing values that were omitted. These 200 missing values might have resulted in the insignificance of P-value. and would I write it in results or discussion?
You should first show that all the data from the 200 record that are "missing" in model D are not relevantly correlated to the outcome. Then you can use the same subset of complete records (all records except those 200) in all your models (A-D).
What is at least as relevant is that the model specifications are sensible. Are the relationshipsbetween the response and the (linear)predictors linear (assuming you used a linear model)? Did you use a sensible distribution model? Did you check residual diagnostics? Did you check how well the models predict the observations? No need to answer my questions. Just to hint that these things are (or should be) considered.
Try running the model AND evaluating VIF (Variance inflation Factors), if the software you are using allows that feature; smaller is better. Rule of thumb for VIF for each factor and interaction is < 2-3.
One caution with MCAR and imputation is that people often impute means when missing data are MCAR because this will not introduce bias. But it will 'artificially' lower the variance of a total. I don't know if that is of any relevance to your application though. It may not be.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence