The dataset I am working with consists of two columns, X and Y where the function Y changes for increasing values of the X variable. If Y shows variations with X, the simplest way to do this is by fitting the data with a linear function. But i am looking for some "advanced" strategy (other than linear regression, i.e., using Neural Networks or other machine learning techniques) to elaborate on these data. The problem is that the data is quite sparse and difficult to directly interpret. So I need a tool/Algorithm/strategy able to catch the patterns/trends hidden in it.
If you're looking to understand how changes in one variable (let's call it X) are connected to variations in another variable (we'll call it Y), there are a couple of techniques you can consider. These techniques can help you spot patterns and connections within your data.
First off, there's correlation analysis: This method lets you gauge how strong and which way the linear relationship between two continuous variables (X and Y) is headed. It gives you a correlation coefficient that gives you an idea of how closely these variables are linked. If this coefficient is close to +1 or -1, it means there's a robust positive or negative linear relationship between X and Y. If it's near 0, that suggests a weak or no linear correlation.
Next up, there's regression analysis: With this approach, you can create a model that captures the connection between X and Y using a regression equation. This equation helps you estimate what the value of Y could be based on the value of X. If you're working with just one independent variable (X) and one dependent variable (Y), that's known as simple linear regression. But if you have more than one independent variable, you're in the territory of multiple linear regression. This technique not only helps you grasp how X and Y are connected but also enables you to make predictions or estimates about Y based on X.
Picking between these two techniques should depend on the type of data you're dealing with and the questions you want to answer. If you're keen on figuring out how strong and which direction the linear relationship is, then correlation analysis is your go-to. On the other hand, if you're aiming to predict or create a model for Y based on X, regression analysis is the way forward. Just remember, though, that correlation doesn't imply causation. Having a high correlation doesn't necessarily mean that changes in X directly cause changes in Y. If you're digging into causation, that might involve designing experiments or diving into more advanced statistical methods.
Lastly, don't forget to check the assumptions that come with the technique you choose. Things like linearity, normality, and homoscedasticity play a role in making sure your analysis is solid and your interpretation of the patterns is sound.
Shagufta Naz Almost all classification algorithms have a regression implementation as well. Maybe you can check this page:
https://scikit-learn.org/stable/supervised_learning.html
You might try this software. It offers much more than linear models, and many real world examples:
www.lerenisplezant.be/fitting.htm
Thank you all for your valuable answers.
I found it very informative to find correlation b/w two variables which was negative (-0.25035), but still I am unable to figure out which kind of technique/algorithm shall I apply to to catch the patterns/trends hidden in two columns of X and Y variable of 1000 rows.
Shagufta Naz First of all, the most important question you should ask, is: what kind of relationship between x and y do you expect, logically, theoretically? THEN choose a model.
If you can give us more details and dat, we can have a look and maybe we will have ideas.
Shagufta Naz The middle part of your graph shows 2 different trends in y. Does the data really oscillate rapidly between distinct high and low values? Or is this the composite of 2 separate runs of data, one persistently low and one persistently high?
Ray Butler no composite. Graph of simple x and y columns mentioned in picture and I need to work on algorithm/techniques which i was asking that can find trend in it when Y changes with respect to x.
Shagufta Naz In that case, with the data oscillating between high and low values, an FFT analysis could reveal the frequency/period of oscillation.
But with no intermediate values between high and low, the result would be affected by undersampling.
And then, on top of the rapid oscillations, you have the slower variation of the high values, with that smooth and prominent peak. You'd have to remove the low values if you wanted to fit something to that trend alone.
You can plot scatterplot for given variables to get pattern of variation with respect to each others. You should check sen slope method to test whether given variable has trend or not if yes, then proceed further according to nature of variable.
Always complex ways aren't the best way. You use scattetplot and acording to it, take the best functions and determin relationship pattern.
- Badges
- Science topic