Saima  -

Holding out some of your data to then see how well you predict for them is helpful.  There are various ways to thus validate. You could research "cross-validation."

Also, the variance of the prediction error is a good indicator.  It assumes you have removed bias (no omitted variables), but if you didn't, it will be inflated, which can give you a good indication anyway.  Your software may have this for you.  For example, STDI in  SAS PROC REG is the square root of the variance of the prediction error.  But it is a good idea to be sure when using a canned software that it is really doing what you think it is doing.

For weighted least squares regression, setting w to the weight in SAS PROC REG would mean w=1/x for the classical ratio estimator, for example, the value for STDI will be different for each y given x. 

I expect this is probably often similar for other software. 

Cheers - Jim

Saima  -

Holding out some of your data to then see how well you predict for them is helpful.  There are various ways to thus validate. You could research "cross-validation."

Also, the variance of the prediction error is a good indicator.  It assumes you have removed bias (no omitted variables), but if you didn't, it will be inflated, which can give you a good indication anyway.  Your software may have this for you.  For example, STDI in  SAS PROC REG is the square root of the variance of the prediction error.  But it is a good idea to be sure when using a canned software that it is really doing what you think it is doing.

For weighted least squares regression, setting w to the weight in SAS PROC REG would mean w=1/x for the classical ratio estimator, for example, the value for STDI will be different for each y given x. 

I expect this is probably often similar for other software. 

Cheers - Jim

Dear Saima,

Descriptive measures of quality of adjustment y are the standard deviations of the estimated parameters, the coefficients of correlation and determination; at the inferential level the quality adjustment is analyzed by t tests and F as well as by the confidence intervals for  prediction (ponctual and on average).

Have a nice time

Helena

Yes, it is by t-test and f-test. These two tests will testify the adequacy of a particular predictive model.

If you are going to use hypothesis tests, you need to be cautious of interpretation.  A p-value does not stand alone, so you need to remember to do a type II error analysis, or some other sensitivity analysis to account for effect size. 

Please see the following:

Press release for the American Statistical Association:

http://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf

My letter in The American Statistician:   

 https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS

Article Practical Interpretation of Hypothesis Tests - letter to the...

Accuracy of prediction models can be assessed by using following metrics: Mean Bias Error (MBE), Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), the fraction of predictions within a Factor of two (FACT2), the Pearson correlation coefficient (R) and the Index of Agreement (IA).

 The MBE provides an indication of whether the predictions are over or under estimated, MAE, RMSE, and FACT2 provide a good indication of how close the modelled and observed values are. A negative MBE value indicates underestimation.

https://www.researchgate.net/publication/267038842_Comparing_the_Performance_of_Statistical_Models_for_Predicting_PM10_Concentrations

Article Comparing the Performance of Statistical Models for Predicti...

Are you looking at predictions or forecasts? 

For example: 

 https://www.researchgate.net/publication/275365705_When_Prediction_is_Not_Time_Series_Forecasting

Research When Prediction is Not Time Series Forecasting

What is the purpose of your prediction?

Suggestions above may be very good but they tend to rely on readily available statistical software. If your purpose is not very well defined that may be the best way to go.

You may want to consider explicitly how you value errors in your prediction.

For example, if you want to cross the street, look left and right to predict the risk of getting run over when you cross now, then you probably put a very different value on the error ‘looks OK and you get run over anyway’ versus ‘let me wait a bit but if you’d cross now you’d arrive on the other side safely’ because you probably value waiting a bit very differently from getting run over.

Or errors quantified as 1 and 2 get valued as 1 and 2 by mean absolute errors respectively, and as 1 and 4 respectively by mean square errors. Is either of those indeed putting the correct values on the errors? E.g. if your are crossing the street with the purpose of catching the bus, then you may be quite indifferent about waiting longer on this side of the street as long as the bus is not coming because crossing sooner just means waiting longer on the other side of the street, but if the longer wait leads to missing the bus, then the value you put on waiting longer is suddenly quite different.

Accurately Measuring Model Prediction Error

When assessing the quality of a model, being able to accurately measure its prediction error is of key importance. Often, however, techniques of measuring error are used that give grossly misleading results. This can lead to the phenomenon of over-fitting where a model may fit the training data very well, but will do a poor job of predicting results for new data not used in model training. Here is an overview of methods to accurately measure model prediction error.

Measuring Error

When building prediction models, the primary goal should be to make a model that most accurately predicts the desired target value for new data. The measure of model error that is used should be one that achieves this goal. In practice, however, many modelers instead report a measure of model error that is based not on the error for new data but instead on the error the very same data that was used to train the model. The use of this incorrect error measure can lead to the selection of an inferior and inaccurate model.

Naturally, any model is highly optimized for the data it was trained on. The expected error the model exhibits on new data will always be higher than that it exhibits on the training data. As example, we could go out and sample 100 people and create a regression model to predict an individual's happiness based on their wealth. We can record the squared error for how well our model does on this training set of a hundred people. If we then sampled a different 100 people from the population and applied our model to this new group of people, the squared error will almost always be higher in this second case.

It is helpful to illustrate this fact with an equation. We can develop a relationship between how well a model predicts on new data (its true prediction error and the thing we really care about) and how well it predicts on the training data (which is what many modelers in fact measure).

True Prediction Error=Training Error+Training OptimismTrue Prediction Error=Training Error+Training Optimism

Here, Training Optimism is basically a measure of how much worse our model does on new data compared to the training data. The more optimistic we are, the better our training error will be compared to what the true error is and the worse our training error will be as an approximation of the true error.

For more plz read at the following link

Best Regards

http://scott.fortmann-roe.com/docs/MeasuringError.html

I recommend the attached article.

The article provides a framework for a number of traditional and relatively novel measures to assess the performance of an existing prediction model, or extensions to a model. Some measures relate to the evaluation of the quality of predictions, including overall performance measures such as explained variation and the Brier score, and measures for discrimination and calibration.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3575184/

(a) R square value (coefficient of determination) will indicate the amount of variance of dependent variable which can be explained by the model from the independent variables.

(b) ANOVA table in output will indicate the statistical significance of the model.   

(c) Role of individual independent variable in relation to the dependent variable can be explained by regression coefficient (beta). Regression coefficient is computed along with its P value. The independent variables having significant P value (P < 0.05) will make the strongest unique contribution in explaining the dependent variable when the variance explained by all other independent variables in the model is controlled for.

ِِِِAssessing the performance of prediction models: a framework for some traditional and novel measures

Ewout W. Steyerberg,1 Andrew J. Vickers,2 Nancy R. Cook,3 Thomas Gerds,4 Mithat Gonen,2 Nancy Obuchowski,5Michael J. Pencina,6 and Michael W. Kattan5

Abstract

The performance of prediction models can be assessed using a variety of different methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic (ROC) curve), and goodness-of-fit statistics for calibration.

Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision–analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions.

We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n=544 for model development, n=273 for external validation).

We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for making clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3575184/

3 Ways to Test the Accuracy of Your Predictive Models

by Victoria Garment 

 In data mining, data scientists use algorithms to identify previously unrecognized patterns and trends hidden within vast amounts of structured and unstructured information. These patterns are used to create predictive models that try to forecast future behavior.

These models have many practical business applications—they help banks decide which customers to approve for loans, and marketers use them to determine which leads to target with campaigns.

But extracting real meaning from data can be challenging. Bad data, flawed processes and the misinterpretation of results can yield false positives and negatives, which can lead to inaccurate conclusions and ill-advised business decisions.

There are many different tests to determine if the predictive models you create are accurate, meaningful representations that will prove valuable to your organization—but which are best? To find out, we spoke to three top data mining experts. Here, we reveal the tests they use to measure their own results, and what makes each test so effective.

Compare Predictive Model Performance Against Random Results With Lift Charts and Decile Tables

Karl Rexer is the founder of Rexer Analytics, a small consulting firm that provides predictive modeling and other analytic solutions to clients such as Deutsche Bank, CVS, Redbox and ADT Security Services. The measures his firm uses to create predictive models are often binary: people either convert on a website or don’t, bank customers close their accounts or leave them open, etc.

Rexer’s firm creates models that help clients determine how likely people are to complete a binary behavior. These models are created with algorithms that typically use historical data pulled from the client’s data warehouse to characterize behaviors and identify patterns.

To test the strength of these models, Rexer’s firm frequently uses lift charts and decile tables, which measure the performance of the model against random guessing, or what the results would be if you didn’t use any model. It’s a methodology commonly used to increase customer response rates by identifying the most promising targets.

Let’s look at an example. One of Rexer’s clients wanted to create a more focused, cost-effective marketing campaign for the coming year, so he and his team built a predictive model for the client using data from the previous year’s campaign.

In the previous year, the client had sent marketing material to 200,000 total leads, 1,579 of which became customers—a conversion rate of 0.8 percent. For the coming year, the client wanted to know in advance which of their new leads would be more likely to buy their product so they could focus their marketing efforts on this segment. Essentially, they wanted a smaller, cheaper campaign with a higher conversion rate.

Rexer and his team first took the client’s lead data and randomly split it into two groups. Approximately 60 percent was used to build the model, while the other 40 percent was set aside to test it (known as the “hold-out” sample).

Splitting the data “helps ensure that you’re not creating a ‘super’ model that only works on that one set of data and nothing else,” Rexer explains. “You’re essentially building something that works on two different sets of data.”

Rexer and his team used a variety of data mining algorithms to comb through hundreds of data fields to find the best set of predictors that worked in the modeling sample. These predictors were used to identify a highly responsive subset of leads.

This algorithm was then used to score each lead in the hold-out sample on a scale of 1 to 100 based upon the characteristics identified by the algorithm, such as how long the person had been a customer and various socio-demographic information. The closer to 100, the more likely the lead was to convert.

The list of leads in the hold-out sample was then rank ordered by score and split into 10 sections, called deciles. The top 10 percent of scores was decile one, the next 10 percent was decile two, and so forth.

For more plz read at following link.

Regards

http://www.plottingsuccess.com/3-predictive-model-accuracy-tests-0114/

 There is an impressive tool to do that easily, you can do k-fold validation or training/testing validation also:

https://www.solver.com/xlminer-data-mining

Hi!

According to your question, MAE is the most natural measure of average error magnitude, and that (unlike RMSE) it is an unambiguous measure of average error magnitude.

For more information, read : Wang, W., & Lu, Y. (2018). Analysis of the Mean Absolute Error (MAE) and the Root Mean Square Error (RMSE) in Assessing Rounding Model. Paper presented at the IOP Conference Series: Materials Science and Engineering.

And

Willmott, C. J., & Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate research, 30(1), 79-82.

Gang Li & Xiaoyan Wang (2018): Prediction Accuracy Measures for a Nonlinear Model and for Right-Censored Time-to-Event Data, Journal of the American Statistical Association, DOI: 10.1080/01621459.2018.1515079

I found also this link helpful for those who use R or Python software

Cross-validation model

https://www.analyticsvidhya.com/blog/2018/05/improve-model-performance-cross-validation-in-python-r/