Keras is a powerful library for deep learning that's easy to use and efficient.

When designing and configuring the deep-learning models, several choices have to be made. Any of these decisions have to be solved empirically by tests and errors and evaluated on actual data.

The high data volume and the complexity of the models take a very long duration of training. As such, it is typically to use a simple separation of data into training and test datasets or training and validation datasets.

For more information, I suggest you follow this link: https://machinelearningmastery.com/evaluate-performance-deep-learning-models-keras/

First: how to select the best configuration for a specific model?

If you mean how to tune hyper-parameters, then there are well-established methods on how to optimize your model's hyper-parameters. Bottom line is, you need to do lots of training runs in a grid search manner.

If you mean architectural modifications (adding/subtracting layers, modifying input/output nodes) it depends heavily on your application and nature of your data.

second: how to compare the different models for the task at hand?

Given the models you chose are good fits to your problem, the best (and pretty much only) way to go is "try and see".

You should consider transfer learning, assuming it is applicable in your case.

Those questions are not silly and the issues you mention apply to all researchers. Unless you can use pre-trained models or apply transfer learning, you will need lots of compute resources and patience to do hundreds of training runs...

- good luck

Relevant objective functions that you can use to evaluate the performance of each model, in a bid to determine the best forecast model/architecture, include:

1) Mean Absolute Error (MAE): Best case = 0.0, Worst case = +ve space

2) Explained Variance Score (EVS): Best case = 1.0, Worst case = 0.0

3) Coefficient of Determination (R^2): Best case = 1.0, Worst case = 0.0

2) Mean Squared Error (MSE): Best case = 0.0, Worst case = +ve space

5) Root Mean Squared Error (RMSE): Best case = 0.0, Worst case = +ve space

Thanks for all of the inputs.

But I think it's still not clear what is the best (practical) practice to conduct a fair comparison for deep learning models given the huge amount of heterogeneous hyperparameters and the randomness in models' nature.

Dear Sarah Almaghrabi ,

Look the link, maybe useful.

https://medium.com/@mahakkothari190.mk/comparison-of-different-deep-learning-models-for-image-classification-1c49f1159d7a

Regards,

Shafagat

Dear Sarah Almaghrabi

Based on my little experience with forecasting problems, the LSTM provides the best near numerical match to the original value.

Parameters that can help you in deciding the best configuration are mostly the same as shared by Bonaventure Molokwu .

I worked with timeseries data so I always used to plot the original and forecasted values (on the same scale/graph) to find the visual match (in small windows first and then increasing it gradually) and it helped; specially if you are a beginner.

I also have a publication that is under review with HadISST Sea Surface Temperature as the parameter under study (as a Book Chapter with Taylor & Francis, CRC Press). Hope to share once it is published.

Please find the following links that may help you:

MATLAB: Long Short-Term Memory (LSTM) Networks - MATLAB & Simulink (mathworks.com)

Python: Predict Stock Prices with LSTM | Python-bloggers (python-bloggers.com)

Thanks and regards

You can use this framework for the comparison of alternative forecasting methods:

Conference Paper Data Formats and Visual Tools for Forecast Evaluation in Cyb...

But firstly, you need to describe your data:

How many time series do you have?

What is the frequency of observations?

What is time series length?

if any work is implemented with a convolutional neural network or any deep learning algorithm, then it compared other deep neural networks like Alex-Net and VGG-network, GoogleNet, and so on.

You can know more about CNN models here. https://journalofbigdata.springeropen.com/articles/10.1186/s40537-021-00444-8

In order to hypertuning, Keras has recently developed and introduced a proper library to adjust the hyper-parameters of ML models and assists you select the optimal set. However, the performance of Keras Tuner depends on several factors.

https://www.tensorflow.org/tutorials/keras/keras_tuner

And also some relevant works are as follows:

Please let me recommend this chapter presenting a detailed workflow for forecasts evaluation and comparison:

Chapter Forecast Evaluation Techniques for I4.0 Systems