Dear Aleksandra Szymura ,

From my experience, working with datasets that include important outliers and extreme values requires a thoughtful approach to normalization, especially when employing autoencoders for dimensionality reduction. To maintain consistency and avoid distortion, the entire dataset should be normalized using its global minimum and maximum values instead of relying solely on the parameters from the training set. This ensures all features are scaled uniformly and prevents issues such as clipping, which can alter critical data points in the test set.

After normalization, applying the trained autoencoder to the full dataset is appropriate as long as the overall data distribution aligns with the training data. A low Mean Absolute Error (MAE) during reconstruction suggests the model generalizes well. Additionally, visual comparisons between the original and reconstructed data can confirm that no significant biases or errors are present.

For clustering tasks like DBSCAN, normalizing the entire dataset prior to dimensionality reduction ensures that feature scales reflect the complete data distribution, which is vital for accurately detecting clusters, including those influenced by outliers. This approach balances effective dimensionality reduction with preserving the essential characteristics of the dataset needed for clustering.

Zimam Ahamed thanks for your answer! But I would like to make dimensionality reduction of one specific dataset. My model does not have to generalize on new, unseen dataset. It is not predictive model. It is only supposed to work for one dataset. It is not typical ML approach...

Maybe I should train and test the model on the same dataset (my full dataset)?

Aleksandra Szymura

Since your model is designed to work specifically with a single dataset and doesn't need to generalize to new data, it's perfectly fine to train and test on the full dataset. To maintain consistency and prevent distortion, you should normalize the entire dataset using its global minimum and maximum values, instead of using the parameters from the training set alone. This ensures that all features are scaled uniformly, avoiding issues like clipping that could distort critical data points.

After normalizing the dataset, you can apply dimensionality reduction using autoencoders. There's no need to split the data into training and testing sets in this case. Once dimensionality reduction is complete, you can proceed with clustering, such as DBSCAN, directly on the full dataset. This approach ensures that outliers and extreme values are handled correctly, and you avoid potential problems that could arise from using a training-test split.

I hope that helps.

Cheers,

Dr. Samer Sarsam

Since your goal is to perform dimensionality reduction on a single dataset without requiring the model to generalize to new data, it is entirely valid to train and test the autoencoder on the same dataset. This approach ensures that the model effectively learns the inherent structure of the data, capturing both typical patterns and outliers, as the focus is not on prediction but on representation.

To ensure consistency, it is important to normalize the dataset using its global minimum and maximum values prior to training the autoencoder. This guarantees uniform scaling across all features and avoids distortions caused by extreme values or outliers. Once normalized, the autoencoder can be trained and applied to the same dataset to produce reduced-dimensional representations.

Evaluating the autoencoder's performance involves assessing the reconstruction quality, particularly for outliers and extreme values, to confirm that critical information has been preserved. While a low Mean Absolute Error (MAE) is a positive indicator, visual comparisons between the original and reconstructed features can provide further assurance of the model's effectiveness.

When proceeding with clustering, such as using DBSCAN, it is important to verify that the reduced dataset aligns well with the algorithm's requirements. Fine-tuning parameters like `eps` and `min_samples` based on the characteristics of the reduced data may be necessary to optimize clustering performance. This structured approach ensures that the dimensionality reduction process meets your specific objectives while retaining the dataset's integrity for further analysis.