It seems to me that you are mixing a few things together. First, the data should reflect the problem you are trying to model. If there are huge shifts in underlying distributions, the model won't be able to exploit learned patterns (look-up concept drift). The best you can do is to extend the data collection process and retrain. Second, the data preparation should start with EDA and implement processing steps based on the results and the requirements of downstream models. You can even couple the data preparation with the models and optimize their hyperparameters together (eg. scaler/transformer selection). Again, this means retraining.

So retrain or use the original scaler with poor results, changing the scaler for prediction is heresy.

PS: MinMaxScaler unifies the feature units and might help with numerical stabilization, does not do much else. If you need a bit more out of your transformations, check this>

https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py

What are your variables and what kind of relationship do you expect?

Md Tohidul Islam

A scaler is a technique for transforming the range of values of a feature or variable in the data. There are different types of scalers, such as MinMaxScaler, StandardScaler, RobustScaler, etc. Each scaler has its own advantages and disadvantages depending on the characteristics and distribution of the data.

The choice of scaler for training and prediction depends on the following factors:

- The range of values in the training data and the prediction data. If the ranges are significantly different, then a scaler that is robust to outliers and extreme values might be preferable, such as RobustScaler or QuantileTransformer¹².

- The distribution of values in the training data and the prediction data. If the distributions are skewed or non-normal, then a scaler that can make them more symmetric or normal-like might be preferable, such as PowerTransformer or Box-Cox Transformer¹².

- The assumptions and requirements of the machine learning model. Some models are sensitive to the scale and variance of the data, such as linear regression, logistic regression, support vector machines, etc. In this case, a scaler that can standardize the data to have zero mean and unit variance might be preferable, such as StandardScaler¹³.

- The interpretability and usability of the scaled data. Some scalers can make the data more difficult to interpret or use, such as log transformation or quantile transformation. In this case, a scaler that can preserve the original range and meaning of the data might be preferable, such as MinMaxScaler¹⁴.

A general recommendation is to use the same scaler for both training and prediction data, and to fit the scaler only on the training data and then apply it to both training and prediction data. This can ensure consistency and avoid data leakage³⁴⁵. However, there might be some situations where using different scalers or fitting them on different data might be necessary or beneficial. For example, if the prediction data has a different distribution or range than the training data, or if there is a need to update or adapt the scaler based on new data⁵.

Therefore, there is no definitive answer to which scaler is better for training and prediction when the ranges are significantly different. It depends on the specific characteristics of the data and the objectives of the machine learning task. A good practice is to experiment with different scalers and evaluate their performance and suitability for the problem at hand.

(1) How to normalize the Train and Test data using MinMaxScaler sklearn. https://stackoverflow.com/questions/50565937/how-to-normalize-the-train-and-test-data-using-minmaxscaler-sklearn.

(2) How to Scale Data for Long Short-Term Memory Networks in Python. https://machinelearningmastery.com/how-to-scale-data-for-long-short-term-memory-networks-in-python/.

(3) Scale Data for Machine Learning - APMonitor. https://apmonitor.com/pds/index.php/Main/ScaleData.

(4) sklearn.preprocessing.MinMaxScaler — scikit-learn 1.2.2 documentation. https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html.

(5) Predicting new data using sklearn after standardizing the training data. https://stackoverflow.com/questions/38780302/predicting-new-data-using-sklearn-after-standardizing-the-training-data.