Thanks Kirill.

My area of application has to do with prediction of output(s) based on given input(s), having trained the system with historical data. So I need the right performance metrics to assess the degree of accuracy of the physical model.

My present work is on path loss modeling. The objective is to establish the relationship between distance, frequency, height, elevation and the output (path loss) over a given area. So, the input and output data set are purely numeric.

It depends on your dataset you are working with. The statistical performance of a model can be measured by many parameters like z-score, t-test etc.  For these statistical analysis first you calculate the outcome like accuracy based on the k-fold cross validation technique. Depending on the outcome calculated like sensitivity, specificity, accuracy etc. in the k-fold, measure the statistical performance. It is one way to check statistical performance of your model. It is widely used in classification problem. If you are working with the continuous output like regression, then you may go for RMSE, MSE etc. According to the your question, your outcome is  to calculate direction means we have fixed no. of outcomes i.e. direction, it is classification problem.  Hope, it will work.

Any of the following statistical performance measure should be appropriate Mean Square Error (MSE), Mean Absolute Error (MAE), Root Mean Squared Error (RMSE).

Hello Segun,

I agree with most suggestions. But we need to know which kind of predictive modelling you are working with and not be assuming. Are you working on regression or classification problem? You said your output is purely numeric (indicative of regression) and another said you may be working on classification based on mentioning direction/path loss as output (but we dont know if this is quantified numerically too or labelled categorically). You know more of your specific research than us even though most people here are in the close area of machine learning for predictive modelling. If you are working on regression, parametric evaluations in the group of MSE, RMSE etc might be adequate. If classification, we can talk of overall accuracy, confusion matrix for computing sensitivity, specificity, etc or the nonparametric use of ROC curves for computing AUC values especially if your data is imbalanced. Another question i have is why are you using machine learning approach for your problem as basic statistical measure in SAS might be able to solve your problem. You have only 4 independent variables right?

Nonetheless, it seems Kirill has done something closer to what you are doing and you may want to give him more clarified details of what you are doing for you to be helped much better. Wishing you all the best.

Adeyemi.

Mean Square Error (MSE),

Mean Absolute Error (MAE),

Root Mean Squared Error (RMSE).