I have several columns of data. In each column the data has different units. Some data is "lower the better" (e.g., cost) and some are "higher the better" (e.,g., employee productivity). To enter data in a model, i need normalized data. How to do this normalization so all values are between 0 and 1.
The goal here is to transform or rescale the numbers that you have from (x_min, x_max) to (0,1).
The formula is new_data = (old_data - x_min) / (x_max - x_min).
The goal here is to transform or rescale the numbers that you have from (x_min, x_max) to (0,1).
The formula is new_data = (old_data - x_min) / (x_max - x_min).
@Kee Huong Lai:
Sir, can we apply it if some data is "lower the better" (e.g., cost) and some are "higher the better" (e.,g., employee productivity)? There is same method to normalize both kinds of data?
First thing as suggested by kee Huong lai.. I agree with his views , in that the values of your variable comes between 0 and 1.
Another approach is that you take the logarithmic of your data set, by this way also most data set can be transformed into normal.
Just check for normallity after applying log transformation by kolmogorov test or Shapiro will test.
I hope it helps
Thanks
For different features/factors, you should use different sets of x_max and x_min. The "lower the better" and "higher the better" data shouldn't be a problem, but if you're using the data, say for multiple regression, these two "lower the data" or "higher the better" will be reflected in the value of the partial coefficients obtained, either positive or negative.
If the data are used as inputs to ANN, sometimes data are normalised to (0,1) and sometimes it's (-1,1), depending on the nature of the classification tasks.
Kee's approach rescales the values. Ravi's suggestion of taking the log will often make the results appear Gaussian. There are other transformations that will also work depending on the distribution of your data (inverse, and square root as two examples). There is a Box-Cox power transformation that might be interesting: https://www.isixsigma.com/tools-templates/normality/making-data-normal-using-box-cox-power-transformation/
The use of normality tests presents a serious philosophical conflict. In introductory statistics most of us are taught that tests are "significant" if the p-value for the test is less than 0.05. Most of us are also taught that the p-value is "the probability of observing an effect as large or larger if the null hypothesis is true." Furthermore, a failure to reject the null hypothesis is not proof that the null hypothesis is true. So the very best that you can hope for from a non-significant test for normality is that the normality test guarantees that the departure from normality of your data are (if present) small enough that it has no significant impact on the reliability of statistical tests that require this assumption. Sadly, I know of no proof that this assertion is true. One must also be concerned that the test for normality was non-significant with my sample size of 30. Yet, this leaves open the possibility that the normality test would be significant if my sample size was 300 or 300,000.
Another thought is that normalizing the independent variables is not central to ordinary least squared regression/ANOVA methods. The requirement for normality is for the residuals.
I am facing the same problem, how to normalise cost (the lower the better) and benefit (the higher the better)?
- Badges
- Science topic
- Similar topics
- Mathematics