I want to analyze the correlation between two variables using 15,000 hourly observations measured simultaneously. When using the raw data, the observed correlation is weak (R² = 0.1). However, after binning the data using the "Sturges grouping method" and performing correlation analysis on the grouped data, the correlation becomes very strong (R² = 0.95).
1. Which method is more appropriate for correlation analysis in this case?
2. Could binning the data lead to overfitting or misinterpretation of the correlation?
3. What is the best approach to determine the true relationship between the variables?
4. Do you think binning the data is a good way to visualise the type of relationship between two variables?
Assuming you're talking about Pearson correlation, you should not bin the raw data. One of the assumptions of the Pearson correlation test is the data are measured on a continuous scale (interval or ratio). If you group the data into bins, they are no longer continuous, but discrete.
One thing that could degrade your correlation is the presence of outliers. I suggest you examine a scatter plot.
Blaine Tomkins, Thank you for your answer. That is a good point you mentioned. I have seen in many publications that people use grouping data with Pearson's correlation analysis. However, I realized that a very high correlation can be observed in the case of grouping regardless of what type of correlation we apply.
David L Morgan Thank you for your answer. Yes, you are right there are significant outliers in my raw data.
Mohammad Alem Sultani Yes, unfortunately many published studies use Pearson's correlation with non-continuous data. If the data are not continuous (i.e., binned), authors should be using Spearman rank-order correlation since binned data are measured on an ordinal scale.
- Badges
- Science topic
- Similar topics
- Higher Education
- Universities