The development of a predictive model is a complex process, which encompass several challenges, assumptions, theoretical implications, and an understanding of the reality of what is being modeled. Models are not just developed because someone feels inspire to develop one. A model is develop from well established theoretical conceptualization of the reality that is to be modeled. With that in mind, the modeler gathers the necessary datasets, which he believes can be used to build the model to predict outcomes that could more likely achieve whatever outcome that is being predicted.
Thus, it is practically irrelevant and repetitive to always build models to predict similar outcome using "good" (quality and quantity) historical data of the reality that is to be modeled. Once a model is developed, it can be applied to new databases or data tables with the similar historical data and context used to develop the model with again an understanding of what is being predicted and what the historical context of the new data looks like.
For example, an auto insurance company developed a predictive model to target new customers who might likely enroll in their auto insurance policy, but wants to score these new customers to know their risks using a range of variables (contain in the model). The insurance company wants to determine a score for each customer, which could be used in the computation of their insurance cost and premium as well as the potential risk that could be predicted and the associated liability.
A predictive model already developed can be applied to these new customers to compute their predictive scores, which represent their levels of risk. In scoring the new dataset, it is important that each variable used in the model development is presented in the new data table that is to be scored. Once the model is applied to the new data table/database, a predictive score of their risks is compute and this score can be used by their system in the determination of the cost of insurance per year. The higher your score the more unlikely you will cause a substantial impact to the insurance company in turns of the number of claims within a given year. The lower your score, the more likely you will file a claim. The scores can be ranked into decile. For example, decile 1 are more preferable than decile 2, than decile 2 to decile 3, etc. Those in the upper 1-2 decile have low insurance cost, while those in the lower decile have high insurance cost.
With that said, there could be so many things wrong with this approach. The assumptions we used both theoretically and in application itself could lead to negative outcomes.
It is very important that we should understand the historical context of the data that we seek to model or that which we seek to apply a model on.
A retail company (let say Best Buy), could decide to develop a predictive model to assess, which catalog campaign circulation have a great performance. The historical context of each campaign needs to be well understood as well as the design, paper, day of the year, season, etc when the catalog was mailed as these could have an impact on the response/performance. Thus, developing a model using the historical circulation data of one catalog campaign and using that model to score other campaigns could yield positive or negative outcomes.
Thus, what do you folks think we need to cautious using a model already developed and applying that model on new datasets with similar variables to predict similar outcome? What we could be doing it wrongly? What do we need to be careful off? What might we want to change, if necessary?
Hello,
I would say that if you are going to use a well established model that was efficient in the past with well known variables (all this should be in methodology so you could replicate the past study) then you know how to build your questionnaire to collect the data.
What seems to me important is the differences regarding characteristics of your sample (potential customers or service users for example). If demographic characteristics are similar for your sample and the past research then you are close to what you need.
Another issue is time frame when this efficient tool was used. As economic market is evolving dynamically, well last two or three years should be acceptable, if more well... I do not know, perhaps the development of new technologies may pose a threat in terms of efficiency of already established psychometric tool.
Regards, Witold
Hello,
I would say that if you are going to use a well established model that was efficient in the past with well known variables (all this should be in methodology so you could replicate the past study) then you know how to build your questionnaire to collect the data.
What seems to me important is the differences regarding characteristics of your sample (potential customers or service users for example). If demographic characteristics are similar for your sample and the past research then you are close to what you need.
Another issue is time frame when this efficient tool was used. As economic market is evolving dynamically, well last two or three years should be acceptable, if more well... I do not know, perhaps the development of new technologies may pose a threat in terms of efficiency of already established psychometric tool.
Regards, Witold
I can't agree more Dr. Witold Orlik. I will provide some contextual response to your answer momentarily.
Evaluate predictive models based on the decisions/actions they empower. As with most tools, predictive models need to be effectively calibrated. This concept may lie at the heart of your question. Regrettably, the devil lies in the detals of all predictive models. As such, I refer you to Douglas Hubbard's classic HOW TO MEASURE ANYTHING in which he outlines the concept of calibration in much more detail than I could ever.
http://www.amazon.com/How-Measure-Anything-Intangibles-Business/dp/1118539273/ref=sr_1_1?s=books&ie=UTF8&qid=1454075015&sr=1-1&keywords=how+to+measure+anything
In order to score new data sets, the data should come from similar that organisation's customers or products and not from other organisations customers or product information. Very often, sister organisations will use the same model in Iceland & Singapore (for example) to predict demand for a soft drink. This is a good example where we cannot use the same predictive model even though it is the same organisation.
Another example, where we need to be cautious is when the input data for the predictive model is very different (in terms of the range of the data inputs, the maximum values, minimum values, etc) from the input data that the model was built on.
Be careful also, not to use predictive models where the new data does not have complete information on all the data inputs necessary to score the new customer accurately.
Hi Colleagues,
All the responses you provided are on point. Let add just few. Assuming you have developed a well-tested, validated, and deployed predictive models and you have conducted post-model performance analysis after the model was deployed into production and facilitate the marketing and segmentation team with selection for their marketing campaigns and you have assess the results of the model in terms of customers response rate (%), dollar per piece, surplus, etc., and you believe, given the results, that the specific model has performed according to the business objectives and you are interested in using the predictive formula (model) to score other customers data file for further campaigning. The below are just few factors you need to have in mind before and during the process.
1. Seasonality: Seasonality in this context means understanding the season when each of your marketing campaigns is launched to your customers. It is during the winter, summer, fall, and or spring? Each customer responds differently when it comes to season in which they are promoted and the product(S) they are targeted with. Thus, ensure that, your new customer data file fits within the season of the model being use. That is, if you are targeting customers in the summer. Make sure that there are recency variables in the model that captures that seasonality. Also, provide to scoring the new customer file, try to do data exploration to understand their seasonal historical context. How have they responded to past campaigns during each season of the year? If you are scoring new customers for a summer marketing campaign, ensure that you subset or segment customers who are more likely to travel during the season.
2. Demographics: Customer demographics are highly predictive of their purchasing and repurchasing propensity. Understanding the demographics of our new customers data table would assist you understand whether or not they should be targeted. One point I would like to make here is that ensure that the demographics of the new customer data file are similar to those upon which the predictive model was trained, tested, and validated to ensure accuracy of the desire response.
3. Data Accuracy: Ensure that the data of the new customer files are accurate. If you input fake and erroneous data into the model you will certainly get a fake and inaccurate result.
4. Predictor/Independent Variables: Ensure that all the predictor or independent variables of the predictive formula are accurately in the new data table. You can ensure that these variables are included by creating them from original data in the new customer data table. If a predictor variable in the predictive formula is missing in the new customer data table, the formula would generate an error message. Your predictive formula would work only when all the independent variables are present in the new customer data table.
5. Do not include the Response or Dependent Variable: When the predictive model (predictive formula) was developed originally, you had a response/dependent/predicted variable as a function of several predictor/independent variables, such that, the regression techniques used captured those to predict the actual vs. the predicted response of a given issue. Thus, once the model (training set) has being tested (tested set) and validated (validation set) and you think statistical details make sense, the resulting predictive formula contains each of the predictor variable and their corresponding parameter estimates along with other statistical details. It is this predictive formula that must be used to score new customers households that fit the desire selection criteria. Thus, the data within the new customers table should not contain the original response variable. In fact, once the predictive formula is used, it should provide score for each customer, which should represent their response.
6. Customer Segmentation: As a Data Scientist, your customers may fall into different segments of your marketing campaign strategy. To address this, ensure that the predictive model that is being used to score new or future customers is sensitive to the various segments of your customers. That is, you should use model segments to score a specialized customer segment. For example, if your company campaigns to an international customer-based, the model you are going to use to score new international customers must be a model that was developed, tested, validated, and assessed in international customers within your organization. If your organization has a marketing campaign for customers in North America, you should use a predictive model specialized for just North American customers. If you have an email campaign vs. a direct mail campaign, those two customers segments behave very differently and should be modeled differently as well. You should not use a predictive model for direct mail campaign to score new customers for an email campaign. That would be suicidal.
7. Customers Loyalty: Customer loyalty is very important for a direct mail and digital marketing standpoint. As stated in # 6, segmentation is a critical element during model design and development as well as during deployment and post-assessment. With that said, the predictive model that is being used to score new customers data tables should be sensitive to their loyalty. Depending on the various loyalty segments of your customers’ profile, you should have separate models for each loyalty segment that is sensitive or specialized to that segment of your customers’ loyalty. Thus, you should not use a predictive model developed for active customers to score new customers who are inquirers. Likewise, you should not use a predictive formula use to reactivate inactive or expired customers to score active customers. The point here is that the predictive model should fit to the audience whose data it seeks to score.
These are some of my thoughts, but there are many.
Note: Always refresh the original model with new data of the original independent variables used to develop the model to keep it updated and conduct series of model discriminant analysis to understand if there is a new independent variable that needs to be developed and or removed.
n
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Analysis
- Data Analysis
- Dataset
- Big Data