What would you consider as the most challenging task developing a predictive model?

Forecast models are an element of creativity and of course should be discussed in a team of professionals. I recommend most often to attend the seminars, symposiums on the subject of your business.

Sérgio Eustáquio Silva

Hello,

As you said it is given a very high emphasis on quality and quantity of data. However, I think, predictive models are first of all a theoretical interpretation of approaching a reality.

So, predictive models are not reality itself but an attempt in advance of trying to build a reality.

All who work with statistics will say that the above statement is so basic that neither need be made l0l but the fact is that in most cases this is not the heart of the matter and not given due consideration in the design of models to think that the most appropriate variables for model building.

The point I want to make is that before worrying about the quality and quantity of variables we must ask ourselves: what variables can help us understand this reality.

So in my opinion always the biggest challenge is the most appropriate design in order to try to capture what variables participate in the construction of a certain reality.

A fact that many despise is that sometimes variables that may seem irrelevant, can have an important role in the model when we think of their predictive ability. Despise several variables "irrelevant" in the model design is not an error so unusual to observe and ultimately may produce biased models and later may prove erroneous or incomplete. Discard variables considered "irrelavantes" also influence the calculation of model errors ...

I agree with our friend Valdimir to know the reality to which will model. It is essential and most often is quite salutary add people who also know that reality on various points of view to an appropriate design of the model.

See you later,

Pierre Souren

Dear Jenkins,

You write: "a lot of emphasis are either placed on

data quality and quantity,

the experience and expertise of the modeler,

the kind of system that is being used to build the model, validate, test, and continue to monitor and assess the model quality and performance over time."

My opinion.

of course the quality of the data (does the variable really represent the construct?) and the quantity (enough degrees of freedom?) are important.

The modeler: of course a more experienced scientist can come up with a better model. But also: a unexperienced scientist is not withhold by knowledge and may come up with creative solutions.

The kind of system, algorithms, software and methods should, of course, match the data and the research question.

I also think that one should differentiate between:

(1)explorative analysis en (2)confirmative analysis.

The challenge with 1 is extracting the important predictors from the set of available predictors and using the best strategies to do so.

For 2 the challenge is the thinking (setting up the model that one will apply when all the data are collected) when setting up a research plan.

Nestor Sherman

All the work that goes into developing the theoretical framework in advance of the analysis. Too many researchers focus on finding statistically significant models and then have a difficult time explaining and justifying their conclusions.

Sérgio Eustáquio Silva

Hello,

Pierre, very good their considerations.

Jenkins, predictive analytics (like almost everything in statistics) are very subtle things and sometimes more important than good predictive models a good analysis of scenarios and people with conditions to have good insights and well understand what the data also say is as important as have good algorithms.

So first think of the scenario (reality) in its multiple factors and interconnections (complexity). After trying to describe this reality. Getting the data (I include here all procedures with collection, verification, reliability, consistency, etc.), analyze the data and try to describe it in a model.

But I would like here to discuss a point. A good analytical model need not be necessarily predictive. Thus, analytical skills can be very useful as predictive models can sometimes not be able to capture something that a well-done analysis sometimes can capture.

I think when we decided to (or need) to work with predictive analytics (sometimes it is the core of our work) improve our analysis models can help improve our predictive models.

Then reconcile these two things is always a challenge, after all our models need to be constantly improved.

See you later,

Andrey Davydenko

One important issue in forecasting is how to evaluate accuracy and compare alternative approaches. Surprisingly, there is no agreement among researchers yet as to what the best error measure is.

We recently conducted a research in order to evaluate the properties of existing error measures and to suggest an improved scheme for measuring accuracy:

https://www.researchgate.net/publication/284947381_Forecast_Error_Measures_Critical_Review_and_Practical_Recommendations

https://www.researchgate.net/publication/282136084_Measuring_Forecasting_Accuracy_Problems_and_Recommendations_by_the_Example_of_SKU-Level_Judgmental_Adjustments

https://www.researchgate.net/publication/257026708_Measuring_Forecasting_Accuracy_The_Case_Of_Judgmental_Adjustments_To_Sku-Level_Demand_Forecasts

The latest publication (Davydenko and Fildes, 2016) summarises findings, whereas earlier publications provide detailed description of the research conducted.

We look at most popular measures (e.g., MAPE, percent better, MdAPE, MASE, GMRAE, etc.) and demonstrate that they simply do not reflect accuracy in terms of a symmetric linear loss and can be rather difficult to interpret (see Section 3 in Davydenko and Fildes, 2016).

When evaluating accuracy at a level of one series only, there are also different views among researchers. We suggest using MAD (or MAE) instead of MSE because of the reasons described in Section 3.1 in (Davydenko and Fildes, 2016).

Article Measuring Forecasting Accuracy: The Case Of Judgmental Adjus...

Chapter Measuring Forecasting Accuracy: Problems and Recommendations...

Chapter Forecast Error Measures: Critical Review and Practical Recommendations

Sérgio Eustáquio Silva

Hello Davydenko,

Congratulations on your research. It was very enlightening everything I read.

See you later,

Andrey Davydenko

Apart from the question of how to measure accuracy, we recently addressed another issue that I think is now becoming important due to limited amounts of available data.

Namely, we researched the question of how to use expert knowledge (managerial expertise/judgmental info) when producing forecasts.

One possible approach here is to fit (estimate) a model based on both actual observations and formalized expert knowledge.

Here's our paper on this topic:

https://www.researchgate.net/publication/282136270_A_joint_Bayesian_forecasting_model_of_judgment_and_observed_data

Technical Report A joint Bayesian forecasting model of judgment and observed data

@Andrey Davydenko (Dr.), all your works look great and I also reason with the highest level of caution about how we as forecaster/modeler can use the unique elements of expert knowledge and judgment in the forecasting and modeling processes. However, given that the judgment and knowledge of an expert could be relative subjective in most instances rather than objective, we should be very cautious on their uses, but the expertise and practical and theoretical judgment are not mutually exclusive of the forecasting and modeling development, deployment, operationalization, management, and continuous assessment.

All the responses here demonstrate great information that we all can benefit from.

Prasanthi Sanjeevi

Assumptions and beliefs have a great influence while developing a predictive model. The challenge is that only some of the assumptions and beliefs are known and too many are unknown (included as ‘Error’). The additional challenge is that real-life data may violate few ‘known ones’. Example: Weather Forecasting models.

Experienced scientist may have a larger set of assumptions and beliefs.

Narasim Ramesh

I am not in this field but the question is interesting and probably provocative.so

two cents worth..IMHO general requirements in prediction is I think:

1. If given an idea query ( examples below) is there a solution ?

1.1 Will robotics be a major market in th next decade?

1.2 Can the present economic model of the world be sustainable?

1.3 Do we need new ethics?

1.4 Will we find another livable planet?

1.5 What are the foreseeable disruptive technologies , social models, political models of the future

2. Given the vast sample space in Big Data how to estimate probability of success of solutions.

3. Is it really possible to always obtain sentient analysis (qualia) using Big Data?

Cheers

Great responses guys. Indeed building predictive models is a complex task, but with the right sets of data, testable assumptions, skills and expertise as well as understanding what it is that you seek to model are all vital components of the model development process.

Gregory Yom Din

The most challenging task is to use the results of the model in decision making. It depends on the research questions, on model formalization and on the data used.

Christopher Lange

The most important part of any predictive model is how well do the assumptions of the model agree with what is actually happening or with how things actually work. One reason Economics is the "dismal science" is that most of its models have no relationship with how economies actually work (e.g., assumption that free trade applies in monopoly capitalism). Another example is in cancer research, i.e., that the cells selected by use of cell surface markers (or combinations of them) are actually the cancer stem cells responsible for recurrence and long-term tumor growth and metastasis. That a few such cells may reside amongst the selected population does not mean that the selected population is composed of (as opposed to containing some) cancer stem cells. One needs to start with simple models that can be tested independently and where each parameter can be individually measured physically. The model also has to make its predictions before the data are collected and analysed, otherwise there is no test.

Dear Christopher! Absolutely. The level of compliance with the constructed model of natural reality.

Arthur Tarasov

Although I haven't finished building my 'big' model yet, I would like to list what I stumbled upon so far:

1. Find good data! Sometimes you need to gather qualitative data too to verify your assumptions. You may need to talk to relevant experts, managers, or ordinary people far far away.... Which brings me to the second issue.

2. Find time and funding. Even though social scientists usually don't need much funding for their experiments, we need a lot of time to put stuff together and the results are every bit as uncertain as in biochem experiments. Producing a good publication may take years with all the reviews you would need to go through. Faculties usually don't want to wait for years and to risk you making a big discovery. You have to be ready to do that though. And to do so it is very important that you...

3. Pick the right topic! I spend more time with my model than I do with my wife and friends combined. You don't have to be so dedicated to be successful but it certainly should motivate you to invest years of your time and, if you succeed, produce a marketable and interesting model that will advance your career and help a lot of people on this planet.

Peer-Olaf Siebers

Building a model that is fit for purpose is one of the biggest challenges in my opinion. With modern technology and big data we can build very complex models, but are they fit for purpose, i.e. helping us to answer the questions we are trying to answer.

I have the feeling that we often build models "because we can" or "because we said we would" but we do not have a well defined purpose for them and when it comes to real world decision making these models are less useful (outside academia).

In my experience with clients, and aside from getting sufficient and accurate data from them with which to quantify the variables relevant to the target prediction, the most challenging aspect of the predictive modeling project is triangulation. So, for example, if using both analytic and simulation methods, the project is not done until the delta in these two predictions is explained. This is where Galilean scientific method comes into play, and what makes this task at once the most challenging, most revealing, and most fun.

Dr. Alberto DiCanzio, I most certainly concur with you that data triangulation is significant in the predictive analytics and modeling spheres as well as any data analyses (quan and qua) might involve. Great point!!!

Jenkins Macedo, the point with which you concur did not surprise you; therefore, I question whether the posting was useful in the context of the goal you stated as "Information provided to this inquiry will be used for personal and professional growth". Possibly this is the case because I was silent on the question about tradeoffs. So, following is my answer to that one. A notable tradeoff with the analytic model I believe to be the characterization of any identifiable stochastic variables versus the cost of determining their probability distribution (which often requires more information than the client has without undertaking research). Resolution of the tradeoff is simple if what is already known about such a variable makes it critical to the accuracy of the prediction. If not so, then one must decide whether a study of the variable does make it critical. If the decision cannot be made by a priori estimation, then it may "satisfice" to take a limited number samples (let's say, eleven or so) of the arrival rate of the random occurrence and estimate its variance simply by Student's t. This too can be checked by repeatedly entering the random variable into the simulation at exponential interarrival rates, varying the parameter lambda. I hope this helps and is not too obvious.

Ron Snee

In my view the most challenging problem is getting quality data; the right data in the right amount at the right time. Also one needs to follow the fundamentals of data-based problem solving as discussed in the following articles:

Snee, R. D., Hoerl, R. W. (2012). Inquiry on pedigree—do you know the

quality and origin of your data? Quality Progress December

2012:66–66.

Snee, R. D., DeVeaux, R. D., Hoerl, R. W. (2014). Follow the fundamentals—

four data analysis basics will help you do big data projects the

right way. Quality Progress, 47:24–28.

“Solving Jigsaw Puzzles: Addressing Large, Complex and Unstructured Problems” (with Alexa DiBenedetto and Roger W. Hoerl), Quality Progress, June 2014, 50-53.

“Applying Statistical Thinking to Big Data Problems” (with R. W. Hoerl and R. R. DeVeaux), Wiley Reviews: Computational Statistics, July/August 2014, 222-232. Cite as: WIREs Comput Stat 2014. doi: 10.1002/wics.1306

@Albert DiCanzio it is not that I am not surprise at what you suggested in relations to data triangulation. It was just simply put did not cross my mind earlier. Thus, I appreciate your comments and post, but that in no way underscore the legitimacy and curiosity stated in the initial post. We all learn and will keep learning until we kick the bucket. Excellent perspectives. I have done data triangulation in several of my field research projects here in the US and abroad and I do know the complexity, challenges as well as potential that that process bears on our results/outcomes.

Your point of data triangulation specific to predictive modeling alerted me to not isolate other complex techniques from statistical analysis in the predictive analytics/modeling sphere.

So, it is not that it didn't contribute to personal growth and development, which in fact it did by reflection.

As a teacher, you don't only learn new ideas, thoughts and concepts, you also re-learn by reflection and your contribution triggered that association.

Does this response sounds fair enough?

Jenkins Macedo, more than fair, in your response I appreciate the didactic perspective on my answers to your good question. To the extent that teaching imparts an understanding of why information is correct or useful, I experience learning as discovering applications of what is taught. In the predictive modeling process, one finds first an understanding of empirical and theoretical information leading to predictive outcomes. Then I see learning from prediction as a step of reconciling the outcomes that lets one verify proposed applications tested in simulation.

@Alberta DiCanzio great and thanks for the prompt responses as well. I am pretty sure our colleagues here on RG are benefitting from all the good information you have provided thus far.

Shah Limon

Besides the quality of data, appropriately considering the variables is one of the most challenging to me. And how Seasonal and environmental factors affects the predicting model !

I wonder what abilities to formal logic was to have a primitive Human to create tools and the first movement (for example, a wheel).

Vladimir's wonderment opens up a potentially fruitful aspect of Jenkins Macedo's question regarding development of predictive models and the relative challenges of tasks incident to that goal. It calls up for consideration the subordinate question whether a prehistoric innovator as represented by the inventor of the wheel had created a predictive model prior to the invention, one that was based on formal logic (or not).

On the one hand, it seems more certain that predictive modeling requires logic than that it requires language. An infant who has not yet learned language is still able to match its parent's behavior to its own in such a way as to predict accurately that certain of its own behaviors will garner the attention of mom or dad. The inventor of the wheel may have first observed that objects which are rotund or have relatively smooth curved surfaces may tend to roll more easily than other objects, and then abstracted this idea (by a combination of observational induction and abstract deduction performed on the objects of induction) into a prediction that a wheel which imitates the shape exhibited by the former class of objects would itself roll easily and act as an efficient carrier if coupled with another carriage mechanism such as a basket.

Yet it is language that, according to Rand [1], is a tool of cognition, one that carries out the integration of perceptual concretes or earlier formed concepts that is required to produce a new concept, like what one might crudely use language to describe as "a carriage" integrated by the coupling of a platform or a box to one or more wheels. Language is thus a tool of conceptual integration, one that can satisfy what Castellano [2] expressed as "the practical need to express logical arguments in human language". His warning about the "failure to distinguish ideas from their representations" would limit any presumption that language (including mathematical language) is necessary to predictive modeling. So, it remains uncertain whether the wheel came before language but if I knew a way of collecting on my bet, I would consider it improbable and bet that language came first.

[1] Ayn Rand, Introduction to Objectivist Epistemology (NY: Meridian, 1990), 10-13.

[2] Daniel J. Castellano, "Logic and Language" http://www.arcaneknowledge.org/philtheo/logiclang/logiclang.htm#s3

Narasim Ramesh

With apologies for deviating from main topic. It is interesting to inquire whether wheels were invented before language.The answer may lie in observation of animal life ( so language is perhaps eliminated( though not communication) . i have observed that dung beetles do in fact make spherical objects which they roll round. (ref below).It leads to further questions. What use is a wheel in isolation without a higher contraption? It is also interesting to note all trees are round in the stem but(open to correction ) no animal life uses the concept of a wheel.

https://www.google.co.in/imgres?imgurl=http://sixlegsphoto.files.wordpress.com/2013/06/anachalcos.jpg&imgrefurl=http://thesmallermajority.com/category/insects/beetles/feed/&h=600&w=900&tbnid=s41cMC1sTXuOqM:&docid=AkCkY5qlrU-QyM&ei=JGGsVuCKGs2QuATf-5PoCA&tbm=isch&ved=0ahUKEwjgrJjq_NDKAhVNCI4KHd_9BI0QMwgnKAwwDA

Cheers

Dear Albert! Dear all! I'm glad I did not break, but rather helped develop such a fruitful discussion. Vladimir

Hi @Vladimir A. Kulchitsky, @Albert DiCanzio, and @Narasim Ramesh (Dr.), I think the theoretical or conceptual point initially triggered by Dr. Vladimir's input warrant a separate discussion that would more likely effectively explore details of the conceptual viewpoints/worldviews brought forward by his contribution that I think might not necessarily to much applicable good to this post. This is not to entirely ignore the relevance of his contribution, but I think it would do more good to have a separate platform to explore that in more details. Does this preposition make sense?