I am using a metric that takes a one time measure, but asks subjects to rate their past behavior numerically, their current behavior numerically, and their intended future behavior numerically.
Since past and current behaviors are not the same as intended behaviors in the future. What is the best way to compare the two?
Additionally, the measure is taking on the day of the treatment (a presentation of risk perception) and is designed as such-
Pre (#) Current (#) Future (#), with an anticipated change between the current and future categories, if the treatment is effective. However comparing current behaviors to future intent seems like a inefficient way to measure effectiveness. The metric is deployed in a quasi-experimental capacity as part of a mandatory program and thus has no control (other than the pre and current.
I have two problems- First, how best (statistically) to get the most out of this design. Second, I may need to redesign the measure entirely (I inherited it from my predecessor), what suggestions do you have about design?
Austin -
I'm not sure I understand the question. If you want to know how some program varies in the way it impacts someone's state of mind, then some sort of regression may be in order, where you may be interested in estimated variance of prediction errors. That is, there may be a useful regression for showing 'future' speculation as a function of current and/or past, but you might want to see how that varies.
However, if you really want to know how well a program is working, you would need to go back to a sample of people who filled out the form in the past, and see what actually happened. This then may still leave you a little short on direct information as to whether the program is working better or worse now than before. But some sort of regression may be useful.
Cheers - Jim
A comparison of apples and oranges occurs when two items or groups of items are compared that cannot be practically compared.
The idiom, comparing apples and oranges, refers to the apparent differences between items which are popularly thought to be incomparable or incommensurable, such as apples and oranges. The idiom may also be used to indicate that a false analogy has been made between two items, such as where an apple is faulted for not being a good orange.
https://en.wikipedia.org/wiki/Apples_and_oranges
Ali, thank you. However, the metric is the tool I have at this moment, and thus must figure out a way to compare previous and current behaviors to future intended behaviors. Though apples and oranges they may be, some comparisons in this case may be drawn, the are both fruit, round etc.
George,
Currently I am looking at the degree of differences between pre current and future answers on the metric, and looking at the direction the answer trend toward. So say a respondent answers their past behavior was a 4, their current behavior is a 3, and their intended future behavior is a 2. I would surmise from this that the intervention did not change the trend the reduction. However, if the respondent rates past behavior at a 3, current behavior at a 3, and future intent as a 2, then there has been some change (as a result of the treatment)
We looked at the problem of how to average relative performance across different objects in these publications:
https://www.researchgate.net/publication/257026708_Measuring_Forecasting_Accuracy_The_Case_Of_Judgmental_Adjustments_To_Sku-Level_Demand_Forecasts
https://www.researchgate.net/publication/284947381_Forecast_Error_Measures_Critical_Review_and_Practical_Recommendations
Article Measuring Forecasting Accuracy: The Case Of Judgmental Adjus...
Chapter Forecast Error Measures: Critical Review and Practical Recommendations
Thank you George, I was firing up the Gaussian process just now. We will see.
Perhaps you also need to ask participants to rate past intentions as well. Then you can rate past intentions with future intentions and past behavior with present behavior. More of an apple to apple (change in intentionality) and orange to orange (change in actual behavior). With the qualification that intentionality is very subjective, as are ratings of one's own behavior.
That is one possibility. Currently I have created statistical filter to isolate any velocity change in intent, which will suffice for now, at least in creating a basic understanding of effectiveness (I have also just established a follow up at 6 months). However, moving forward, it is my intent to rebuild the measure. I think analyzing previous intent in valuable. I think shifting to target those not commonly associated with "high risk" is value to isolate as to what the presentation effect is on them. And finally, really nailing down the idea that high risk is relatively inverted, and risk perception plays a key role in risk behaviors. What we suspect is that many risk prevention programs are targeted toward a small pct of population whom are not necessarily the ones who wind up in trouble, injured, and so forth. Bringing this into better focus will be key.
- Badges
- Science topic