I have a dataset which three variables: date, mean1 and mean2.
date - datetimens64 (YYYY-mm-dd format)
mean1 - float type
mean2 - float type
The data is not normally distributed. I want to find the impact of date on mean1 and mean2. And as I read though I learnt that date could be an independent variable and mean1 and mean2 could be dependent variables (on date). Therefore, the goal is to whether independent variable has any impact on dependent variables therefore I went through regression models and found that my scenario falls under multi-target / multivariate regression model. But I am not sure if I can use date as the independent variable. Could you please suggest something? I also thought of running linear regression twice: 1) between date and mean1, 2) between date and mean2. But I had to convert date to integer first.
I would rather prefer to test both the dependent variables together though, and also I want to keep the independent variable as date itself.
Any suggestion please?
NOTE: i am working on python (pandas dataframe). It would be helpful if you could share some similar implementations.
When the data is normally distribution you can used paired t-test (if a few samples) or used z-transformation and also approximately CLT when the data is a huge number. If the data is not normally distribution and the kind of your data is ordinal or interval you can used wilcoxon test. CMIIW.
You can possibly filter the dataframe using the date and do a two-sample Wilcoxon test (one-tailed if you can to see if one of the means is better than the other or two-tailed if you just want to check if the means are different). You might want to use scipy.stats.wilcoxon() function in python for this where you can indicate greater or less for the one-tailed test!
Hello Samiksha,
Your data are counts/frequencies. It seems to me that the simplest test would be something like a chi-square goodness of fit test (before freq vs. after freq.). Any departure from the expected values of 50% of total freq for each occasion would suggest an inequality.
No distributional assumptions required. It is true that, as the total N (frequency) increases in sheer size, the power of the test increases as well.
If you do choose to apply the Wilcoxon test (as suggested by others, though I wouldn't recommend it), be sure that this is the Mann-Whitney-Wilcoxon test (for independent samples), as there is no good way to match the data points before vs. after for a Wilcoxon signed-rank sum test. Note that the MWW requires homogeneity of variance. You'd also have to assume that each day's result is an independent observation. As well, the "power" of such a test will not be great unless you collect a fair number of days' worth of data before and after the event, unless the magnitude of the effect is substantial.
Good luck with your work.
David Morse thank you the detailed explanation.
Like you said:
As well, the "power" of such a test will not be great unless you collect a fair number of days' worth of data before and after the event, unless the magnitude of the effect is substantial.
Well, I will be considering different time spans, i.e, 3 days before and after the event, 7 days before and after the event, 15 days before and after the event. This is to check which time span gives the best results.
Also I have a doubt whether I should use autocorrelation or any regression model (linear or binomial) where I can check if the event has an impact on the count (or mean of those counts)?
Dear Samiksha Samiksha ,
A paired samples t-test is the suitable technique for comparing the means of two samples when each observation in one sample can be paired with an observation in the other sample. Since you are using Python, import and use the scipy.stats package (import scipy.stats as stats), and apply the wilcoxon () function (since your data is not normally distributed) to the two arrays you created which contain pre-event production counts (array "a" - sample of observations of production counts before the event ) and post-event production counts (array "b" sample of observations of production counts after the event). The Wilcoxon signed-rank test does not assume normality in the data, hence it can be applied when the assumption of normality has been violated.
- Badges
- Science topic
- Similar topics
- Analytical Chemistry
- Column