Hi,
I'm trying to merge two chlorophyll datasets from different sensors by applying linear regression on their data overlap. The journal that I'm following log transformed (base 10) the data from both datasets before applying regression, and then used the resulting coefficients to correct one of the datasets. This means I have to log transform all my data to create my merged time series. However, I am hoping to use untransformed chlorophyll in the end.
My questions are (1) is it necessary to log transform the data in this situation, or just directly input untransformed data for regression? (2) If I do use the coefficients from the log-log regression, how should I apply it to the uncorrected dataset? Thank you very much!
Justin -
Could you be using logs to reduce heteroscedasticity?
It sounds like you want to use the overlap in data sets to see how one predicts for the other, and then assume that relationship holds up outside of that set? Hopefully you have reason to think that. If you have enough "overlap" data, perhaps you can see if you get compatible results when you do this for the first half of these "overlap" data, and then for the second half, considering the estimated variances of the prediction errors, or at least the standard errors of the slopes.
At any rate, to consider heteroscedasticity, I would avoid any transformations and use weighted least squares regression, considering heteroscedasticity in the error structure. The first link below provides some examples, and the next is for a tool you can use to do this.
https://www.researchgate.net/publication/333642828_Estimating_the_Coefficient_of_Heteroscedasticity
https://www.researchgate.net/publication/333659087_Tool_for_estimating_coefficient_of_heteroscedasticityxlsx
However, it sounds like you should also look into serial correlation. That isn't an area I can help you with though. I've worked more with finite populations than time series. I suggest you might want to research "generalized least squares" (GLS regression).
Best wishes - Jim
Muhammad Ali Is correct. Always back transform your results to get your units right from your prediction equation if you predict something. Best, David Booth
Using logs needs care, because it is a non-linear transformation and changes the biases in your data. This may or may not be advantageous. A linear correlation in the log domain is not a linear correlation in the ordinary domain.
The ordinary correlation is fitting the equation Y = a +bX to the data. If instead you fit log Y = a + b log X, that is the same as fitting Y = a.Y^b .
If you are just trying to merge data rather than compute statistics, you may want to try both. Alternatively, you may look for the linear transformation that results in both datasets having the same mean and variance. If there isn't such a transformation, the datasets may not be mergeable.
The anti-log of the arithmetic mean of logarithms is not the arithmetic mean of the data!
Personally, I feel it is always a good idea to start with the original (raw) data since you do not know exactly what was done before analysis such as removing "outliers", weighting, etc,. Many sins are hidden in pre-analysis "cleanup" so as to reinforce the preconceived notion of the researchers. Also, look at any discrded data to see what if anything it "says".
If this is for a class or research in the college environment, consult your advisor/professor for a final judgement
The obvious alternative to what was suggested is to just construct a single overall combined data set with "sensor type" as a two-valued basic dummy variable. By constructing extra dummy variables (from this basic one) you could then go on to see/test if the sensor-type has any effect on the individual regression coefficients. But you may really need to start by thinking about what type of effect you think might occur on the observed value from your two types of sensors.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics