Hello,
Say I have a target metric and a second metric related to the target, and I want to measure the effect of an intervention which affects both metrics, but not always equally. The target metric has higher variance than the related metric, but the related metric only has correlation of ~0.5 to the target metric. After the intervention, should I measure the effect using the target metric or the related metric?
I could not find any literature on this common problem. I attempted the problem myself, using simulation to produce time series similar to the data, and adding a percentage increase to the target and related metric. Then I would use DID to make an estimate of the increase. Defining the increase that should be added to the related metric is difficult, as the increase must reflect not only the relationship between them, but also how likely it is that an intervention will change the relationship. I chose to add the same percentage increase to both target and related metric, with a standard error depending on the time-varying correlation between them. This produced reasonable results (for low sample sizes, related metric has lowest MAPE, then at some point target becomes better), but my methods were somewhat arbitrary.
- What existing research is there on measuring impact analysis when multiple metrics are available?
- What are some approaches you think could work?
- Badges
- Science topic
- Similar topics
- Publishing