After initial preprocessing like missing value & outliers imputation I found the data is still noisy. and as to denoise the data I used SMA. and then on SMA data points I did the linear regression to find out the slop to use in prediction.
is it valid to use linear regression on SMA data?
Note: I applied fourier transform and low pass filter to denoise the data but the p-value of the residuals come out to be 0.
The standard errors will be too small. The linear regression *is* a de-noising technique (anything except a linear trend is considered noise and eliminated in the fitted line). So if you think that fitting a linear trend makes sense, you could use the linear regression to de-noise, and then fit a linear regression to the de-noised data: you will get identical estimates but all standard errors will be zero (it's the extreme case of what I meant in the beginning). I hope you see that this makes not much sense.
If you think that the relationships are not linear, so that a SMA will better "follow the data", then the relevant question is: why would you use linear regression? So this would make no sense either.
I would say NO, for the following reasons,
- if your data is real, and from a random sample, then it will always have noise.
- The moving average is developed to understand the fluctuation of the data in a specific "time" window and that its behavior (up, down or no variation) can be associated with a certain lag.
if your data corresponds to a time series, your forecasts would be more accurate with AR, MA, ARMA, ARIMA, ARCH, GARCH models, and you can make very good predictions. When trying to build the linear regression model it is likely that the autocorrelation assumption is violated, generating a problem that some do not take it into account and others correct it by altering the response. BUT, if your data are cross-sectional, everything Jochen explained will make sense.
Why would you do that? The regression itself is a "denoising". Always use the raw data for regression! Any pre-filtering can incluence the regression an an unpredictible way.
The SMA is a smoothing technique that helps remove noise and highlight underlying trends in a time series. It calculates the average of a specific number of previous data points, usually in a sliding window fashion. The choice of the window size will impact the denoising effect; smaller window sizes may preserve more noise, while larger window sizes may oversmooth the data and potentially remove important information.
Linear regression, on the other hand, assumes a linear relationship between predictor variables and the response variable. It estimates the slope and intercept of a linear equation that best fits the data points. Linear regression assumes that the residuals (i.e., the differences between the observed and predicted values) follow a normal distribution, have constant variance, and are independent of each other.
Overall, using SMA as a denoising technique and subsequently applying linear regression is a valid approach, but it is essential to interpret the results carefully and assess the assumptions and limitations of both techniques in your specific context.
Regression analysis can play a certain role in noise reduction, and preprocessing is also very important, depending on the specific situation of the sample for corresponding preprocessing.
The regression analysis does not require preprocessing of data but testing of its assumption is prerequisite process for fitting the regression model so if assumptions are get satisfied then do regression directly withaout determining simple moving averages
I have done the same data preprocessing. I have 5000 records, each has 5 inputs and 3 outputs. I am trying to train a Neural Network model to predict the outputs. When using raw data for training, the model performance is disappointing. However, using a movemean average increased the accuracy of the model significantly. I am wondering if this data preprocessing is valid?
- Badges
- Science topic
- Similar topics
- Psychology